Multi-person picture synthesis method and device, equipment and medium

Abstract

The invention discloses a multi-person picture synthesis method and device, equipment and a medium. The method comprises the following steps: splicing original images at multiple angles to obtain a panoramic image of a current target area; real-time tracking is carried out according to all face position information in the current frame, sound source information is collected synchronously, and a current spokesman in the target area is determined; and generating a multi-person synthetic picture according to the real-time motion picture of the current spokesman and the real-time motion pictures of other people. Multi-angle images are collected for feature point matching and image splicing, a high-quality panoramic image is generated, wide-area visual angle coverage is achieved, a more detailed image is obtained, all face positions are locked, corresponding sound source information is collected, a current spokesman is recognized, a multi-person synthetic picture which highlights the current spokesman and gives consideration to other people is generated, and the multi-person synthetic picture is obtained. The integrity of the panoramic information is guaranteed, meanwhile, the key content is clearly presented, the problems that the visual angle is limited and information is overloaded are solved, and the communication efficiency and watching experience of a teleconference are improved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent conferencing equipment and computer vision technology, and in particular to a method, apparatus, device, and medium for multi-person image synthesis. Background Technology

[0002] With the popularization of remote work and distributed collaboration models, video conferencing systems have become a core tool for daily communication in enterprises and organizations, and their image capture and presentation effects directly affect the efficiency of meeting communication.

[0003] Currently, mainstream conference cameras are mainly divided into two categories: traditional monocular or binocular fixed-view cameras and multi-view panoramic cameras. Traditional monocular or binocular cameras suffer from limited field-of-view coverage; while multi-view panoramic cameras can achieve 360-degree coverage, they can only perform simple stitching and fusion of the original images, resulting in unclear details in the fused image and affecting the remote viewing experience. Furthermore, the large size of the fused image makes it difficult to focus on key areas, failing to meet the core needs of efficient meeting communication.

[0004] Therefore, in multi-person meeting scenarios, existing conference camera systems suffer from problems such as incomplete field of view coverage, poor dynamic adaptability, and unsatisfactory image presentation, which seriously affect the communication efficiency and viewing experience of remote meetings. Summary of the Invention

[0005] Therefore, it is necessary to propose methods, devices, equipment, and media for multi-person video synthesis to address the above-mentioned problems, so as to improve the communication efficiency and viewing experience of video conferencing.

[0006] To achieve the above objectives, the first aspect of this application provides a method for compositing multiple person images, the method comprising: A multi-view camera is used to acquire original images of the target area from multiple angles in real time, and the original images are preprocessed to obtain a processed standard image. Feature point matching and image stitching are performed based on feature points in all the standard images to obtain a panoramic image of the current target area. Based on the preset face recognition technology, face detection is performed on the panoramic image to determine the position information of all faces in the current frame; Based on the position information of all faces in the current frame, real-time tracking and synchronous acquisition of sound source information are performed to generate real-time audio signals of the target area and real-time motion images of each face. Based on the real-time audio signal and the real-time motion image, sound source localization and voice stimulation are performed to determine the current speaker in the target area; Based on preset layout rules, a multi-person composite image is generated according to the real-time motion images of the current speaker and other people.

[0007] Furthermore, the step of performing feature point matching and image stitching based on feature points in all the standard images to obtain a panoramic image of the current target region specifically includes: Based on the scale-invariant feature transform algorithm, feature points in all the current standard images are detected to obtain feature point information corresponding to each standard image, wherein the target standard image is any one of the standard images; Based on the K-nearest neighbor matching strategy, feature point matching is performed according to the feature point information corresponding to all the standard images to obtain an initial set of feature point matching pairs. Using the random sampling consensus algorithm, mismatched pairs in the initial feature point matching pair set are removed to obtain the remaining target feature point matching pair set. Based on the target feature point matching set and the standard image, the images are stitched together to obtain a panoramic image of the current target area.

[0008] Furthermore, the step of stitching together the target feature point matching set and the standard image to obtain a panoramic image of the current target region specifically includes: Based on the target feature point matching, the set and the standard image are spatially aligned to determine the overlapping area of ​​the standard images corresponding to adjacent lenses of the multi-camera after the alignment. Based on the linear brightness adjustment algorithm, the brightness is adjusted according to the brightness of the overlapping area of ​​the standard images corresponding to the adjacent lenses to obtain the adjusted target image; According to the multi-resolution fusion algorithm, the overlapping areas of the target images corresponding to adjacent shots are fused in layers to obtain a panoramic image of the current target area.

[0009] Furthermore, the step of performing face detection on the panoramic image based on preset face recognition technology to determine the position information of all faces in the current frame specifically includes: The panoramic image is input into a preset YOLOv8 network structure to obtain several candidate face bounding boxes and several candidate face key point sets output by the YOLOv8 network structure. The candidate face bounding boxes include the border position and confidence of the first candidate face, and the candidate face key point sets include the key point positions of the second candidate face. Based on the nonmaximum suppression algorithm, according to the face bounding box position and confidence of each first candidate face, redundant bounding boxes with an overlap higher than a preset overlap threshold are removed from all candidate face bounding boxes to obtain the remaining effective face bounding boxes. Pose estimation is performed based on the facial key point positions of each second candidate face to determine the pose information of the second candidate face; The effective set of facial key points is determined by analyzing the pose information of each second candidate face and the preset angle threshold. Based on all the valid face bounding boxes and the set of valid face key points, determine the position information of all faces in the current frame.

[0010] Furthermore, the step of real-time tracking and synchronous acquisition of sound source information based on the position information of all faces in the current frame to generate real-time audio signals for the target area and real-time motion images of each face specifically includes: Using the Kalman filter algorithm, based on the face position information of the target face in the current frame and the obtained state information of the target face, the predicted position region of the target face in the panoramic image of the next frame is predicted, wherein the target face is any one of all faces in the current frame; Based on a preset target tracking algorithm, face tracking is performed in the predicted location area of ​​the target face to determine the real-time position of the target face in the panoramic image of the next frame. Based on the real-time position of the target face, a rectangular image region centered on the face is cropped from the panoramic image of the next frame to generate a real-time motion picture of the target face. The size of the rectangular image region is preset and its area is smaller than the area of ​​the panoramic image. The raw audio signal of the target area is acquired in real time using a preset multi-microphone array; The original audio signal of the target area is denoised and enhanced to generate a preprocessed real-time audio signal.

[0011] Furthermore, the step of determining the current speaker within the target area by performing sound source localization and voice stimulation based on the real-time audio signal and the real-time motion image specifically includes: Based on the speech activity detection algorithm, noise filtering is performed on the real-time audio signal corresponding to each microphone in the multi-microphone array to obtain a target real-time audio signal containing only valid speech signals. Synchronize and align the target real-time audio signal corresponding to each microphone to determine the start time characteristics of the effective speech signal; A timestamp set is generated based on the timestamp of the target real-time audio signal corresponding to each microphone, according to the start time characteristics of the effective voice signal; Determine the set of time differences between all microphones in the multi-microphone array based on the set of timestamps; The directional information of the effective speech signal is obtained by performing least squares calculation based on the time difference set and the position information of each microphone in the multi-microphone array. Confidence analysis is performed on the real-time motion images of each face to determine the tracking confidence level for each face. Based on the location information, face tracking confidence level, and real-time motion images of each face, the current speaker within the target area is determined.

[0012] Furthermore, determining the current speaker within the target area based on the location information, face tracking confidence level, and real-time motion images of each face specifically includes: Based on the distance between the location information and the center of the face in each real-time motion frame, a set of candidate speakers is determined, as well as the distance deviation value between the location information and each candidate speaker; The distance deviation value and the face tracking confidence of each candidate speaker are weighted and summed to obtain a comprehensive index for each candidate speaker. The candidate speaker with the highest comprehensive index is the current speaker in the target area.

[0013] To achieve the above objectives, a second aspect of this application provides a multi-person image synthesis apparatus, the apparatus comprising: The image acquisition module is used to acquire original images of the target area from multiple angles in real time using a multi-view camera, and preprocess the original images to obtain processed standard images; based on the feature points in all the standard images, feature point matching and image stitching are performed to obtain a panoramic image of the current target area; The face tracking module is used to perform face detection on the panoramic image based on preset face recognition technology, determine the position information of all faces in the current frame, and perform real-time tracking and synchronously collect sound source information based on the position information of all faces in the current frame to generate real-time audio signals of the target area and real-time motion images of each face. The image synthesis module is used to perform sound source localization and voice stimulation based on the real-time audio signal and the real-time motion image, and determine the current speaker in the target area; based on preset layout rules, it generates a multi-person composite image based on the real-time motion image of the current speaker and the real-time motion images of other people.

[0014] To achieve the above objectives, a third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described in the first aspect.

[0016] The present invention has the following beneficial effects: This invention proposes a method for multi-person image synthesis, comprising: using a multi-camera array to acquire original images of a target area from multiple angles in real time, and preprocessing the original images to obtain processed standard images; performing feature point matching and image stitching based on feature points in all standard images to obtain a panoramic image of the current target area; performing face detection on the panoramic image based on preset face recognition technology to determine the position information of all faces in the current frame; performing real-time tracking and synchronously acquiring sound source information based on the position information of all faces in the current frame to generate real-time audio signals of the target area and real-time motion images of each face; performing sound source localization and voice stimulation based on the real-time audio signals and real-time motion images to determine the current speaker in the target area; and generating a multi-person composite image based on preset layout rules and the real-time motion images of the current speaker and other people. This invention generates high-quality panoramic images by performing feature point matching and image stitching on multi-camera arrays to acquire images from multiple angles in real time, achieving wide-area view coverage and obtaining more detailed images. Combining facial recognition and real-time tracking technologies, the system simultaneously locks the location of all faces and collects corresponding audio source information. Utilizing sound source localization and voice stimulation, it intelligently identifies the current speaker and dynamically generates a multi-person composite image that highlights the current speaker while also taking into account other participants, thereby enhancing the dynamic adaptability of the meeting scenario. It can automatically focus on the current speaker, ensuring the integrity of panoramic information while clearly presenting key content. This effectively solves the problems of limited viewing angles and information overload in panoramic images associated with traditional devices, improving the communication efficiency and viewing experience of remote meetings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] in: Figure 1 This is a flowchart illustrating the multi-person image synthesis method in an embodiment of the present invention; Figure 2 This is a structural block diagram of the multi-person image synthesis device in an embodiment of the present invention. Figure 3 This is an internal structural diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To address the problems of incomplete field of view coverage, inaccurate image focus, poor dynamic adaptability, and insufficient audio-visual coordination in existing conference camera systems, this invention proposes an intelligent image synthesis system and method that integrates multi-view panoramic stitching, face recognition, face tracking, and voice stimulation technologies. This system achieves 360-degree coverage of the conference scene, accurate identification and dynamic tracking of participants, adaptive image layout based on the number of participants, and automatic focus on the speaker, thereby improving the communication efficiency and viewing experience of remote conferences.

[0021] For details, please refer to [link / reference]. Figure 1 , Figure 1 This is a flowchart illustrating a multi-person video composition method according to an embodiment of the present invention. The method can be applied to multi-person video conferencing scenarios and includes: Step 100: Use a multi-view camera to acquire original images of the target area from multiple angles in real time, and preprocess the original images to obtain the processed standard images.

[0022] In this embodiment, the target area can be a conference room, that is, a place where multiple people participate in a meeting. Multi-view cameras are used to capture original images of the target area from multiple angles in real time. These original images are then preprocessed to obtain a processed standard image, which is then stitched together.

[0023] In one feasible embodiment, a multi-camera is a camera containing multiple lenses. It is understood that the more lenses a camera has, the more content it can capture, resulting in a wider coverage area and achieving 360-degree field-of-view coverage to accurately capture every person within the target area. Preferably, the multi-camera can be a quad-camera, which incorporates four 1 / 2.82-inch CMOS sensors and four single-field-of-view (HFOV / VFOV) ultra-wide-angle fixed-focus lenses of 117° / 62° to acquire raw image data covering 360°.

[0024] In one feasible embodiment, preprocessing of the original image may include image noise reduction, correction, color equalization, and other processing.

[0025] Step 200: Perform feature point matching and image stitching based on the feature points in all standard images to obtain a panoramic image of the current target area.

[0026] In this embodiment, parallel feature extraction is performed on all acquired standard images. A feature description algorithm with rotation invariance and scale invariance is used to accurately detect and describe feature points in each standard image. These feature points typically correspond to textured areas in a conference room scene, such as corners, table edges, decorations, or the edges of stationary objects. After extracting feature points, feature point matching is performed between multiple images, and the images are stitched together based on the matching results. A panoramic image of the target area in the current frame is obtained, providing a complete scene base map for subsequent personnel detection and tracking.

[0027] Step 300: Based on the preset face recognition technology, perform face detection on the panoramic image to determine the position information of all faces in the current frame.

[0028] In this embodiment, the face recognition technology is a pre-trained lightweight deep convolutional neural network model. This model has been trained on a large-scale face dataset and possesses high-precision face detection and key point localization capabilities, adapting to complex lighting backgrounds and multi-person occlusion scenarios in conference rooms. The panoramic image generated in step 200 is used as input data, and the network simultaneously captures detailed information of large-sized faces in the foreground and small-sized faces in the background, thereby solving the problem of facial scale differences caused by varying distances between attendees and the camera. By accurately pinpointing the specific facial positions of all attendees within the current frame of the panoramic image, reliable target data is provided for subsequent face tracking and identity recognition.

[0029] Step 400: Track and collect sound source information in real time based on the position information of all faces in the current frame, and generate real-time audio signals of the target area and real-time motion images of each face.

[0030] In this embodiment, based on the acquired face position information in the current frame, the system performs real-time dynamic tracking of all participants in the panoramic image to ensure that the movement trajectory of each face is recorded continuously. At the same time, the system simultaneously starts the audio acquisition module, uses a microphone array to pick up and beamform the sound signals in the target area in real time, and generates real-time audio signals of the target area, providing accurate audio-visual source data for subsequent judgment of the speaker's status and synthesis of intelligent meeting view.

[0031] Step 500: Based on the real-time audio signal and real-time motion image, perform sound source localization and voice stimulation to determine the current speaker in the target area.

[0032] In this embodiment, the real-time audio signal generated in step 400 is subjected to spectrum analysis to accurately calculate the sound source location information and achieve sound source localization. In order to further eliminate environmental noise interference, the system also performs voice activity detection and energy analysis on the audio signal, calculates the sound energy intensity and duration in a specific frequency band, and determines that there is a valid speech at a certain location when the audio energy value at a certain location continuously exceeds the preset voice excitation threshold. Combining the acquired face and its real-time coordinate information in the panoramic image, the sound source location coordinates corresponding to the valid speech are matched with the spatial positions of all participants in real time to determine the face corresponding to the sound source and thus determine the current speaker.

[0033] By comprehensively comparing and logically correlating the spatial location of the sound source, the characteristics of the voice energy, and the real-time tracking of facial motion, it is possible to accurately identify and lock onto the target person who is currently speaking, and determine them as the current speaker, providing a precise decision-making basis for subsequent focus switching and intelligent layout generation.

[0034] Step 600: Based on the preset layout rules, generate a multi-person composite image according to the real-time motion images of the current speaker and other people.

[0035] In this embodiment, the system first selects an appropriate screen segmentation mode based on the preset layout rules and the number of faces detected in the current frame. For example, when only one person is speaking and there are multiple participants, a large-screen close-up mode of the speaker is used to highlight the key points; when multiple people are participating at the same time and no one is speaking, the system automatically switches to picture-in-picture, split-screen, or grid arrangement mode.

[0036] By prioritizing the real-time motion of the current speaker during the generation of the composite image, the system extracts clear details of the speaker's face using digital zoom or cropping algorithms and dynamically allocates them to the main view area or prominent position of the composite image, ensuring that the speaker's expression and demeanor are presented first. Simultaneously, the system adaptively scales and adjusts the real-time motion of non-speakers (other participants), filling the remaining secondary view areas according to layout rules, thus constructing a multi-person view with a clear hierarchy.

[0037] The multi-person video synthesis method proposed in this invention combines panoramic view coverage with intelligent focusing. It utilizes multi-view panoramic stitching technology to construct a seamless meeting scene base map, effectively solving the problem of missed participants caused by the limited field of view of traditional cameras and ensuring the complete presentation of all participants. Simultaneously, by tracking faces and locating sound sources, it accurately captures the current speaker and adaptively adjusts the screen layout based on the number of participants. This not only ensures that the speaker's image is always in the best viewing state, enhancing the sense of presence and focus in remote communication, but also avoids the tediousness and delays of manual operation through intelligent screen switching and synthesis, significantly enhancing the dynamic adaptability and interactive experience of video conferencing.

[0038] In one embodiment of the present invention, step 100, preprocessing the original image to obtain a processed standard image, specifically includes: performing image correction, image noise reduction and color equalization on the original image to obtain a processed standard image.

[0039] Specifically, firstly, the ISP (Image Signal Processor) processing unit applies a fisheye distortion correction algorithm to address the barrel distortion produced by the ultra-wide-angle lens, performing geometric transformations using pre-calibrated lens distortion parameters. The fisheye image is then projected onto a plane to ensure that straight lines remain straight.

[0040] Secondly, an algorithm combining two-dimensional Gaussian filtering and median filtering is adopted to process the salt-and-pepper noise and Gaussian noise in the original image in layers. The salt-and-pepper noise is removed by median filtering, and the Gaussian noise is smoothed by Gaussian filtering. At the same time, key details such as facial contours and facial features in the image are preserved to avoid excessive blurring.

[0041] Finally, based on the grayscale world algorithm, the average RGB channel values ​​of the original images captured by multiple lenses are calculated, and the gain and offset of each channel are adjusted to ensure that the images captured by different lenses are consistent in color temperature, brightness, and saturation, thus avoiding obvious color banding after stitching.

[0042] This invention employs a pre-deployed multi-camera array to synchronously acquire raw images, obtaining a 360-degree panoramic view of the target area. By effectively eliminating geometric distortion caused by the ultra-wide-angle lenses in the multi-camera array through the ISP processing unit, it ensures straight lines and accurate spatial relationship restoration, significantly improving the alignment accuracy and geometric consistency of subsequent panoramic stitching and avoiding stitching misalignment or ghosting caused by distortion. Simultaneously, a layered noise reduction strategy combining two-dimensional Gaussian filtering and median filtering is used. This efficiently filters out image interference such as salt-and-pepper noise and Gaussian noise while preserving the clarity and texture information of key details such as facial contours and features to the greatest extent possible. This solves the problem of detail loss or excessive blurring that can easily occur with single noise reduction algorithms, providing high signal-to-noise ratio and high-quality basic image data for subsequent face recognition and tracking. In addition, the color balance processing based on the grayscale world algorithm achieves uniformity in color temperature, brightness and saturation of multi-lens images, effectively overcoming the color inconsistency caused by differences in the perspectives of different sensors and lenses, eliminating color breaks and abruptness in the panoramic stitching and fusion area, ensuring the color continuity and naturalness of the final composite image, thereby greatly improving the overall visual experience and immersive experience of remote conferencing.

[0043] In one embodiment of the present invention, step 200, performing feature point matching and image stitching based on feature points in all standard images to obtain a panoramic image of the current target area, specifically includes: S210. Based on the scale-invariant feature transform algorithm, feature points in all current standard images are detected to obtain feature point information corresponding to each standard image. The target standard image is any one of all standard images.

[0044] In this embodiment, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract feature points from the preprocessed standard images of multiple shots. By constructing a Gaussian difference pyramid, extreme points in the image are detected, and the position, scale, and orientation of the feature points are determined to ensure the stability of the feature points under different scales and rotation angles.

[0045] S220. Based on the K-nearest neighbor matching strategy, feature point matching is performed according to the feature point information corresponding to all standard images to obtain an initial set of feature point matching pairs.

[0046] In this embodiment, one standard image is used as a reference image, the feature point descriptors of the reference image are used as the query set, and the feature point descriptors of the remaining images are used as the target set. The similarity of feature points is measured by calculating the Euclidean distance between feature descriptors. For each feature point in the reference image, the two nearest neighbor and the second nearest neighbor are searched in the target set. If the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a preset ratio threshold, the nearest neighbor feature point is determined to be a valid matching point of the corresponding feature point in the reference image, thereby eliminating possible fuzzy matching and mismatch. By traversing all standard images and repeating the above process, a set of initial feature point matching pairs with high reliability is selected. These feature point pairs verified by the ratio test are integrated to construct an initial set of feature point matching pairs containing the correspondence between pairs of images, providing accurate registration data for subsequent image transformation matrix calculation.

[0047] To better understand, the following example will be used to illustrate the point: Suppose there are two standard images to be stitched together. Image A has m feature points, and image B has n feature points. Calculate the Euclidean distance between each feature point in image A and each feature point in image B, and sort the results. Select the two feature points b and c in image B that are closest to feature point a in image A as its nearest and second nearest neighbors. If the ratio of the distance from feature point b to feature point a to the distance from feature point c to feature point a is less than a preset threshold (e.g., 0.75), the matched feature point is considered valid, and feature point a and its nearest neighbors are considered an initial feature point pair.

[0048] S230. Using the random sampling consensus algorithm, remove mismatched pairs from the initial feature point matching pair set to obtain the remaining target feature point matching pair set.

[0049] In this embodiment, the homography matrix between standard images is calculated iteratively using the Random Sample Consensus Algorithm (RANSAC) to further eliminate mismatched pairs caused by similar textures or repetitive structures in the initial feature point matching pair set.

[0050] In each iteration, the random sampling consensus algorithm randomly selects a small number of feature point pairs (e.g., four feature point matching pairs) from the initial set. Using these samples, a preliminary perspective transformation matrix is ​​estimated. This matrix is ​​then used to transform the feature points of a standard image, and the projection error between the transformed points and the matching points on other standard images is calculated. If the error is less than a preset error threshold, the feature point is classified as an interior point. By repeating this process iteratively, the target perspective transformation matrix with the most interior points is retained. The interior point set is determined using the target perspective transformation matrix, and feature point pairs that do not conform to the optimal model parameters are classified as exterior points (i.e., mismatches) and removed. Finally, only feature point matching pairs that have passed the verification of the target perspective transformation matrix and possess high geometric consistency are retained, generating the remaining set of target feature point matching pairs. This significantly improves the robustness and accuracy of image registration.

[0051] S240. Based on the target feature point matching, the set and the standard image are stitched together to obtain a panoramic image of the current target area.

[0052] In this embodiment, the target perspective transformation matrix and the feature point matching pairs in the target feature point matching pair set obtained in step S230 are used to determine the coordinate mapping relationship between the standard images to be stitched together, and a unified output panoramic canvas coordinate system is established accordingly. Multiple standard images are projected onto the panoramic canvas coordinate system according to the mapping relationship. The overlapping and non-overlapping areas of each image on the panoramic canvas are calculated. The overlapping areas are fused, while the original image pixels are directly retained for the non-overlapping areas to ensure the integrity of the image information. Finally, all the projected and fused pixel data are integrated to generate a panoramic image of the current target area with high resolution, wide field of view and no visual abrupt changes, so as to achieve accurate 360-degree all-round reproduction of the meeting scene.

[0053] Specifically, S240, based on target feature point matching, stitches the set and standard images together to obtain a panoramic image of the current target region, including: S241. Based on the target feature point matching, the spatial position of the set and the standard image is aligned, and the overlapping area of ​​the standard image corresponding to the adjacent lenses of the multi-view camera is determined after the position alignment.

[0054] In this embodiment, the target perspective transformation matrix calculated in step S230 and the feature point matching pairs in the target feature point matching pair set are used to construct the spatial mapping relationship from each standard image to the panoramic canvas coordinate system. Matrix operations are used to transform the pixel coordinates of each standard image to a unified coordinate system, thereby achieving precise spatial alignment of multiple standard images. During this process, based on the field of view settings and calibration parameters between adjacent shots, the horizontal and vertical projection ranges of adjacent standard images on the panoramic canvas are calculated, and the geometric regions where these projection ranges intersect are determined. These intersecting regions are identified as overlapping regions. Simultaneously, validity detection is performed on non-overlapping parts to remove image edge holes or invalid pixels caused by perspective transformation, thereby accurately defining the overlapping regions of the standard images corresponding to adjacent shots during the stitching process, providing an accurate pixel processing range for subsequent brightness adjustment and image fusion.

[0055] S242. Based on the linear brightness adjustment algorithm, the brightness is adjusted according to the brightness of the overlapping area of ​​the standard images corresponding to adjacent shots to obtain the adjusted target image.

[0056] In this embodiment, the average brightness value of the overlapping area of ​​two adjacent standard images is calculated, and the brightness difference coefficient between the images is obtained by comparing these two average brightness values. Subsequently, based on the difference coefficient, a linear transformation model is used to perform gain compensation on the pixel values ​​of the image to be adjusted. That is, the pixel points in the overlapping area are adjusted point by point using the brightness difference coefficient and the offset, so that the average brightness of the adjusted image in the overlapping area is close to that of the reference image, thereby eliminating the brightness jump at the stitching seam caused by the difference in exposure parameters of different lenses or sensor characteristics. After completing the brightness statistics and parameter calculation of the overlapping area, the linear adjustment model is extended to the global range of the image to be adjusted, and the brightness of the entire image is uniformly corrected to generate the adjusted target image, thereby ensuring that multiple images have a consistent brightness visual effect both globally and locally, laying the lighting foundation for subsequent image fusion.

[0057] In one feasible embodiment, the brightness difference coefficient k is obtained as follows: The average brightness value of two adjacent standard images within the overlapping region is calculated respectively. Let the average brightness of image A in the overlapping region be μ. A The average brightness of image B in the same overlapping region is μ B Average brightness is typically calculated using the arithmetic mean of grayscale values ​​or pixel values ​​(0-255). The brightness difference coefficient is determined based on the ratio of the average brightness of image A to that of image B. Optionally, the brightness difference coefficient k can be calculated using the following formula: K=μ B / μ A In one feasible embodiment, the offset is calculated using the following formula: B=μ B -(k×μ A ) S243. Based on the multi-resolution fusion algorithm, the overlapping areas of the target images corresponding to adjacent shots are fused in layers to obtain a panoramic image of the current target area.

[0058] In this embodiment, the Laplacian pyramid decomposition technique is used to decompose adjacent target images after brightness adjustment into hierarchical structures of different frequency bands, including a bottom layer image that retains only low-frequency information and a top layer image that retains edge details and high-frequency textures at each level. For each level of the image, a corresponding weighted coefficient mask is constructed in the overlapping area according to the distance weight of each pixel from its respective image center or edge. This mask is used to perform weighted fusion of the images at each level, so that a smooth transition of lighting and background is achieved in the bottom layer image, while clear and sharp edge details and texture information are retained in the top layer image. Subsequently, the fused multi-level pyramid image is reconstructed and inversely transformed layer by layer, that is, starting from the top pyramid, it is sampled down and fused with the next level until the original image resolution is restored. Finally, a seamless and detailed panoramic image of the current target area is generated, thereby effectively eliminating stitching marks and improving the visual consistency of the overall image.

[0059] This invention, through a comprehensive image processing workflow based on scale-invariant feature transformation, K-nearest neighbor matching, random sampling consensus algorithm, and multi-resolution linear brightness adjustment and fusion, fundamentally eliminates image inconsistencies caused by optical lens distortion, sensor exposure differences, and installation angle deviations in multi-view cameras. It ensures sub-pixel level geometric alignment between adjacent images, effectively avoiding common issues like ghosting, misalignment, and discontinuity in panoramic stitching. Simultaneously, by utilizing brightness statistics and linear compensation strategies in overlapping areas, the system achieves dynamic balance between global and local brightness across multiple images. Combined with a multi-resolution fusion algorithm, it smooths background lighting while preserving high-frequency texture details, completely eliminating visual imperfections such as stitching seams and ghosting. The resulting panoramic image possesses high definition, a wide field of view, and highly consistent color and brightness, realistically and completely reproducing the details of a 360-degree conference scene. This provides high-quality base map data for subsequent face detection, tracking, and intelligent image synthesis, significantly improving the visual presentation of the remote conferencing system.

[0060] In one embodiment of the present invention, step 300, based on a preset face recognition technology, performs face detection on the panoramic image to determine the position information of all faces in the current frame, specifically including: S310. Input the panoramic image into the preset YOLOv8 network structure to obtain several candidate face bounding boxes and several candidate face key point sets output by the YOLOv8 network structure. The candidate face bounding boxes include the border position and confidence of the first candidate face, and the candidate face key point sets include the key point positions of the second candidate face.

[0061] In this embodiment, the panoramic image of the current frame of the target area is resized to fit the input requirements of the YOLOv8 network structure. The feature extraction network and attention mechanism integrated in the YOLOv8 network structure are used to perform deep feature extraction on the input adjusted panoramic image, thereby accurately capturing the visual features of different faces in the complex indoor background of the target area.

[0062] The network structure includes a face boundary regression branch and a keypoint regression branch. The two branches output several first candidate face bounding boxes containing location coordinates, width and height dimensions, and target confidence, as well as several second candidate face keypoint sets containing corresponding facial features such as eyes, nose tip, and mouth corners, and confidence. This process can simultaneously handle the detection needs of large-sized faces in the foreground and small-sized faces in the background of panoramic images, effectively solving the problem of face scale differences caused by the different distances of attendees from the camera, and providing high-precision, multi-dimensional basic detection data for subsequent face tracking and identity recognition.

[0063] S320. Based on the non-maximum suppression algorithm, according to the face bounding box position and confidence of each first candidate face, remove redundant bounding boxes with an overlap higher than a preset overlap threshold in all candidate face bounding boxes to obtain the remaining valid face bounding boxes.

[0064] In this embodiment, firstly, all bounding boxes of the first candidate faces are sorted in descending order according to their corresponding confidence scores. The bounding box with the highest confidence score is selected as the baseline target and retained as a valid face bounding box. Secondly, the intersection-over-union ratio (IoU) between each pair of valid candidate face bounding boxes is calculated. This ratio reflects the degree of overlap between the two bounding boxes. The calculated IoU is compared with a preset overlap threshold. If the IoU is greater than the preset overlap threshold, the face bounding boxes with lower confidence scores are removed, thereby effectively removing invalid boxes that are repeatedly detected for the same face. By iteratively repeating the above filtering process in the remaining candidate face bounding box set until all bounding boxes have been traversed, a set of valid face bounding boxes with no repetitions and the highest confidence score is finally selected. This not only eliminates the redundant interference caused by multi-scale detection but also significantly improves the accuracy and uniqueness of face localization.

[0065] S330. Based on the facial landmark positions of each second candidate face, perform pose estimation to determine the pose information of the second candidate face.

[0066] In this embodiment, using the set of key facial points extracted in step S310, specific key points that can characterize facial structural features are selected, such as the coordinates of five key facial features: the center of the eyes, the tip of the nose, and the left and right corners of the mouth, to construct a geometric model of facial pose. By calculating the relative positional relationship of these key points in the image plane coordinate system, and combining the camera imaging principle and multi-view geometric constraints, the yaw angle, pitch angle, and roll angle of the face relative to the camera optical axis in three-dimensional space are calculated, thereby accurately determining the head pose information of the second candidate. This process can not only quantify the orientation and rotation angle of the face, but also effectively distinguish extreme poses such as frontal, side, and back-to-the-camera poses, providing key geometric judgment criteria for subsequent screening of effective faces suitable for identity recognition.

[0067] S340. Analyze the pose information of each second candidate face and the preset angle threshold to determine the set of valid facial key points.

[0068] In this embodiment, based on the pose information of the second candidate face, the yaw angle, pitch angle, and roll angle included in the pose information are compared one by one with preset angle thresholds to determine whether the face orientation meets the recognition requirements for a frontal view or a small-angle side view. If the angle value of any dimension exceeds the preset angle threshold range, the candidate face is determined to be in an extreme posture such as a large-angle side turn, head down, or head up. Such candidate faces with severely obscured or distorted facial features due to excessive posture will be regarded as invalid data and removed. Conversely, if the angle values ​​of all dimensions are within the preset angle threshold range, the candidate face posture is determined to meet the recognition standard, and the corresponding candidate face key point set is classified into a valid face key point set. This accurately filters out highly usable and feature-complete frontal face data from complex panoramic scenes, significantly reducing the error rate of subsequent face recognition tasks.

[0069] S350. Based on the set of all valid face bounding boxes and valid face key points, determine the position information of all faces in the current frame.

[0070] In this embodiment, the set of valid facial key points that meet the pose criteria selected in step S340 is integrated with the set of valid facial bounding boxes output in step S320 to determine the position information of all faces in the current frame.

[0071] In a feasible embodiment, precise matching can be performed based on the set of effective facial key points and the set of effective facial bounding boxes. The coordinates of the key points are bound to their respective bounding boxes according to the spatial correspondence between the two. For each effective target, a complete facial location information including the center coordinates of the face, boundary size, and pose features is comprehensively constructed by combining the global location range provided by the bounding box and the local facial feature positions provided by the key points. The system summarizes all the filtered facial location information in the current frame according to a preset index or priority order to generate a final list of face detection results for the current frame. This list not only accurately eliminates false detections and targets with abnormal poses, but also retains high-precision positioning details, thereby providing accurate and reliable data support for subsequent face tracking and intelligent speaker analysis.

[0072] In another feasible embodiment, the effective set of facial key points and the effective set of facial bounding boxes can be integrated, and the union of the two sets can be used as the position information of all faces in the current frame, thereby avoiding missed detections caused by postures such as people looking down or turning their faces to the side.

[0073] This invention improves the robustness and accuracy of face detection in complex panoramic meeting scenarios by introducing the YOLOv8 network structure and multi-dimensional pose analysis. The non-maximum suppression algorithm effectively solves the problems of duplicate bounding boxes and redundant interference in multi-scale detection, ensuring the uniqueness and accuracy of face localization. Simultaneously, through pose estimation and angle threshold filtering based on keypoint sets, the system can automatically eliminate low-quality faces unsuitable for feature recognition, such as those with large-angle side profiles or heads tilted down, accurately locking onto frontal faces that meet recognition requirements. This multi-level processing strategy, from coarse-grained bounding box detection to fine-grained pose filtering, not only optimizes the quality of detection data and eliminates invalid interference, but also provides a high-purity, highly usable location information foundation for subsequent face feature extraction, identity matching, and intelligent tracking, thereby significantly improving the recognition efficiency and accuracy of the entire intelligent video conferencing system.

[0074] In one embodiment of the present invention, after determining the position information of all faces in the current frame, the faces are counted, and a maximum detection limit of p faces is set. When the number of detected faces exceeds p, the p faces with the highest confidence are retained first; when the number of detected faces is less than p, the actual number of detected faces is counted, and the information of the areas where no faces were detected is recorded to provide a basis for subsequent image layout.

[0075] In one embodiment of the present invention, step 400, which involves real-time tracking and synchronous acquisition of sound source information based on the position information of all faces in the current frame, to generate real-time audio signals for the target area and real-time motion images of each face, specifically includes: S410. Using the Kalman filter algorithm, based on the face position information of the target face in the current frame and the obtained state information of the target face, predict the predicted position region of the target face in the panoramic image of the next frame, where the target face is any one of all faces in the current frame.

[0076] In this embodiment, target tracking is performed on each face present in the current frame. For example, any target face among all faces in the current frame is tracked. Based on the target face's center coordinates and motion speed in the panoramic image, a Kalman filter state equation and observation equation are established. The motion trajectory of the target face at future moments is calculated using the Kalman filter state equation and observation equation, and then linearly recursively estimated. In the prediction phase, the predicted position region of the target face in the panoramic image at the next moment is derived using the face position and state information from the previous moment. At the same time, the uncertainty of the prediction result is quantified using the process noise covariance matrix, generating a confidence region containing the predicted coordinates and the prediction error range. This process not only smooths the position changes of the face based on its historical motion trajectory, but also provides accurate search priors through the prediction mechanism when the face is briefly occluded or there is a delay in image frame processing, ensuring the speed and robustness of target tracking in the next frame.

[0077] S420: Based on a preset target tracking algorithm, perform face tracking in the predicted location area of ​​the target face to determine the real-time position of the target face in the panoramic image of the next frame.

[0078] In this embodiment, the predicted location region of the target face is used as the search window, significantly narrowing the pixel range for face matching in the panoramic image. A preset target tracking algorithm is used to perform refined feature matching and position correction within this region. By extracting the visual features of the target face and calculating the similarity with the image content within the search window, the feature matching similarity is obtained. Combined with the distance weight of the predicted location center, the optimal matching position of the target face in the next frame of the panoramic image is locked. After determining the real-time position, the actual observed position coordinates are fed back to the Kalman filter as new observation values ​​to correct and update the state prediction model. Thus, even when the target face is displaced, deformed, or partially occluded, the tracking continuity and high accuracy can still be maintained, achieving stable tracking of the faces of participants in dynamic conference scenarios. In one feasible embodiment, the target tracking algorithm is a KCF model. When a face is detected to be partially occluded, for example, when the occlusion area is ≤30%, the state prediction by Kalman filtering is matched with the historical features of the KCF model to maintain the tracking state. When the occlusion area is too large, for example, when the occlusion area exceeds 30% and the duration is ≤1s, the tracking ID is retained, and accurate tracking is resumed after the occlusion is removed. When the occlusion area is too large and the occlusion duration exceeds 1s, the tracking ID is released and face detection is performed again.

[0079] In one feasible embodiment, when multiple tracking targets are located close to each other (e.g., overlap ≥ 20%), the target tracking confusion is avoided by adjusting the search window size and kernel function parameters of the KCF algorithm, ensuring that the tracking ID of each target is unique and stable.

[0080] S430. Based on the real-time position of the target face, a rectangular image region centered on the face is cropped from the panoramic image of the next frame to generate a real-time motion picture of the target face. The size of the rectangular image region is preset and its area is smaller than the area of ​​the panoramic image.

[0081] In this embodiment, based on the real-time position of the target face, the center point of the target face is used as the cropping reference. The corresponding pixel area is locked in the next frame of the panoramic image, and a rectangular image area is constructed by expanding outward according to the preset length and width dimensions to ensure that the face is always in the visual center of the image and to reserve appropriate movement margin. During the cropping process, the edge coordinates of the rectangular image area are monitored in real time. Once the cropping box is detected to exceed the physical boundary of the panoramic image, the cropping center is automatically adjusted or an edge filling strategy is adopted to avoid black borders or information loss caused by image truncation. Finally, the cropped rectangular image area is decoded and encapsulated to generate a real-time motion image of the target face that is independent of the panoramic view and contains clear facial details. This image not only removes redundant background interference, but also provides a standardized video data stream for subsequent independent split-screen display and facial detail analysis.

[0082] S440 uses a preset multi-microphone array to collect the original audio signal of the target area in real time.

[0083] In this embodiment, a pre-defined multi-microphone array is deployed within the target area. By utilizing the different spatial distribution positions of the microphones, a beamforming pickup pattern with specific directionality is formed, thereby enabling real-time acquisition of the sound field signal in the conference room from all directions. During the acquisition process, the multi-microphone array synchronously receives sound wave vibrations from different directions and converts them into analog electrical signals. Subsequently, a high-precision analog-to-digital converter samples and quantizes the analog electrical signals into digital time-domain signals, generating an original audio data stream containing the voices of all participants and environmental background noise. This process can preserve the spectral characteristics of the voice and the location information of the sound source to the greatest extent. Through the redundancy and complementarity of multi-channel signals, the signal-to-noise ratio and dynamic range of the audio acquisition are effectively improved, laying the signal foundation for subsequent sound source localization, noise reduction processing, and high-fidelity voice restoration.

[0084] In one feasible embodiment, a built-in array of six omnidirectional digital MEMS microphones can be used, with a microphone spacing of 5cm and a hexagonal distribution to ensure uniform audio signal acquisition within a 360° range. Each microphone supports a 24-bit / 48kHz sampling rate and a signal-to-noise ratio ≥60dB, improving the quality of voice signal acquisition.

[0085] S450: Denoise and enhance the original audio signal of the target area to generate a preprocessed real-time audio signal.

[0086] In this embodiment, the acquired raw audio signal undergoes spectral analysis. Adaptive filtering algorithms or spectral subtraction are used to accurately identify and suppress steady-state environmental noise and sudden interference noise, effectively improving the signal-to-noise ratio of the speech signal. Subsequently, a speech enhancement algorithm is used to dynamically equalize the denoised signal, automatically compensating for and enhancing the characteristic components of the human voice frequency band, while repairing speech distortion that may be caused by noise suppression, making the sound clearer and more natural. The processed audio signal undergoes echo cancellation and automatic gain control to further eliminate reverberation interference and stabilize volume fluctuations, ultimately generating a preprocessed real-time audio signal with a clean background and prominent speech, thus providing high-quality audio data input for subsequent speech recognition and sound source localization analysis.

[0087] This invention combines a Kalman filter prediction mechanism with a target tracking algorithm to achieve accurate and stable face tracking in conference scenarios. It ensures continuous and high-precision face location even under conditions of target occlusion, posture changes, and personnel movement. Simultaneously, it utilizes a multi-microphone array for high-fidelity audio acquisition, followed by denoising and speech enhancement to reduce environmental noise interference and generate clear and pure speech signals, significantly improving sound quality. This prediction-based adaptive cropping mechanism can output stable and centered real-time motion images in complex panoramic scenes, optimizing video data availability and achieving deep fusion of facial visual information and high-quality auditory information. This provides reliable and accurate multimodal data support for subsequent sound source localization, speaker identification, and intelligent conference decision-making.

[0088] In one embodiment of the present invention, step 500, which involves performing sound source localization and voice stimulation based on real-time audio signals and real-time motion images to determine the current speaker within the target area, specifically includes: S510: Based on the speech activity detection algorithm, noise filtering is performed on the real-time audio signal corresponding to each microphone in the multi-microphone array to obtain the target real-time audio signal containing only valid speech signals.

[0089] In this embodiment, preliminary endpoint detection is performed using acoustic feature parameters such as short-time energy and zero-crossing rate to distinguish speech segments from non-speech silence segments in the audio stream, thereby extracting potential speech activity intervals. Subsequently, a Gaussian mixture model or a deep neural network model is applied to the signals within these intervals for refined noise assessment, accurately identifying and separating steady-state background noise and environmental interference components (e.g., coughing, sighing, environmental noise, etc.), filtering out non-speech noise components from the original signal. Finally, the target real-time audio signal with only effective speech components and a significantly improved signal-to-noise ratio is output. This process effectively eliminates the influence of environmental background noise on subsequent time delay estimation, ensuring the accuracy and robustness of the sound source localization algorithm.

[0090] S520: Synchronize and align the target real-time audio signal corresponding to each microphone to determine the start time characteristics of the effective voice signal.

[0091] In this embodiment, based on a preset time reference and sampling rate, the real-time audio signals of each target channel acquired by the multi-microphone array are time-synchronized and aligned to correct phase deviations caused by differences in hardware acquisition circuits or signal transmission delays. Waveform envelope analysis and / or frequency domain feature mutation detection are performed on the aligned audio data to capture the mutation point where the effective speech signal first breaks through the energy threshold in each channel. The sampling time corresponding to this mutation point is then analyzed as the start time feature of the effective speech signal. This process eliminates time inconsistencies between multi-channel signals, ensuring that the start time used for subsequent delay calculations has high time synchronization and feature consistency, thus providing a reliable data foundation for accurately quantifying the minute time differences in sound wave arrival at different microphones.

[0092] S530. Generate a timestamp set based on the timestamps on the target real-time audio signal corresponding to each microphone, according to the start time characteristics of the effective speech signal.

[0093] In this embodiment, for each microphone channel in the multi-microphone array, based on the start time feature determined in step S520, the specific sampling point index of this feature in the corresponding audio data stream is found and extracted, and converted into a precise timestamp value. Subsequently, the timestamp values ​​extracted from all microphone channels are arranged and integrated in order of microphone number to construct a timestamp set containing complete timing information. This set not only records the order in which sound waves arrive at each microphone unit in the array, but also quantifies the time difference of the sound source signal during spatial propagation in the form of time data, providing a standardized numerical basis for subsequent calculation of the time difference between microphones, thereby ensuring the accuracy of sound source localization calculation.

[0094] S540. Determine the time difference set between all microphones in the multi-microphone array based on the timestamp set.

[0095] In this embodiment, based on the time values ​​of each channel recorded in the timestamp set, the reception time of any microphone channel is used as a reference. The timestamps of all other microphone channels are then subtracted from this reference value to calculate the relative delay time of the sound wave signal arriving at different microphone units in the array. By traversing all possible channel combinations in the microphone array, these pairwise relative delay times are aggregated and summarized to construct a complete time difference set characterizing the differences in sound wave propagation paths in space. This time difference set eliminates the influence of the system's absolute time and retains only the relative time offset reflecting the directional characteristics of the sound source, effectively improving the analytical accuracy of sound source spatial location.

[0096] S550: Perform least squares calculation based on the time difference set and the position information of each microphone in the multi-microphone array to obtain the directional information of the effective speech signal.

[0097] In this embodiment, based on the known geometric topology of the multi-microphone array, the precise spatial position information of each microphone in a preset coordinate system is obtained. The time difference set is then substituted into the hyperboloid model for sound source localization, establishing a set of nonlinear observation equations relating the spatial coordinates of the sound source to the time difference of arrival. Considering the noise and errors present in actual measurements, a least squares algorithm is used to numerically solve the observation equations. By constructing an objective function and iteratively optimizing, the optimal solution that minimizes the sum of squared residuals between the predicted and measured time differences is found, thus fitting the optimal spatial position coordinates of the sound source. Finally, these three-dimensional spatial coordinates are converted into azimuth angle and distance information relative to the target area, generating accurate azimuth information of the effective speech signal indicating the current direction of sound emission. This azimuth information can include azimuth and elevation angles, effectively eliminating the interference of random errors on the localization results and achieving precise locking of the speaker's position.

[0098] S560: Perform confidence analysis based on the real-time motion images of each face to determine the tracking confidence for each face.

[0099] In this embodiment, multi-dimensional feature extraction and state evaluation are performed on the real-time motion images of each face. By comprehensively quantifying key indicators such as face detection confidence, feature matching similarity, and position change continuity, a fusion computing model is constructed to output the confidence score of each tracking target. The numerical range of this score can be set between 0 and 1.

[0100] Specifically, during the calculation process, the face confidence score, which represents the probability of face features detected by S310, is used as the basic detection confidence score. The cosine similarity between the face features of the current frame and those of historical frames is calculated using a feature extraction model (such as ArcFace or MobileFaceNet) to obtain the feature matching similarity score, which measures the consistency of the target identity. A high similarity score indicates that the current frame and historical frames belong to the same target, while a low similarity score may indicate that the target has been lost or replaced. The smoothness of the movement trajectory of the face center coordinates between consecutive frames is analyzed to measure the continuity of positional changes. Finally, the normalized face detection confidence, feature matching similarity, and position change continuity are weighted and fused to obtain a comprehensive confidence value. This comprehensive confidence value is then compared with a preset confidence threshold. When the comprehensive confidence value is greater than or equal to the confidence threshold, the corresponding face is determined to be in a stable tracking state and directly participates in subsequent speaker identification. When the comprehensive confidence value is lower than the confidence threshold, it is determined to be unstable tracking or suspected loss, thereby automatically triggering a re-detection mechanism to correct the tracking state and ensure that the visual basis for current speaker identification is accurate and reliable.

[0101] In one feasible embodiment, the feature matching similarity of the target face is calculated using the following formula:

[0102] In the formula, The feature vector of the target face in the current frame. It is the average value of the feature vectors of the target face in the first N frames.

[0103] S570: Based on location information, face tracking confidence level, and real-time motion images of each face, determine the current speaker within the target area.

[0104] In this embodiment, the sound source orientation information is mapped onto the panoramic image coordinate system of the target area to determine the projection position of the sound source on the image plane or the corresponding orientation sector, which serves as the initial geospatial reference for screening. Subsequently, for all faces in a stable tracking state, the Euclidean distance or angular deviation between the sound source projection position and the real-time center coordinates of each face is calculated to initially screen one or more face candidates that are closest to the sound source and within the effective sound pickup range. Based on this, the selected candidate faces are weighted and evaluated in conjunction with the tracking confidence generated by S560, and targets with high tracking confidence are selected as the final matching objects to eliminate low-quality tracking interference caused by occlusion or motion blur. Finally, the face that meets the sound source orientation matching and has the highest tracking confidence is locked as the current speaker, thereby realizing the deep fusion and judgment of auditory spatial location information and visual target state information, effectively solving the problem of speaker misidentification in multi-person overlapping scenarios.

[0105] Furthermore, the S570, based on location information, face tracking confidence level, and real-time motion images of each face, determines the current speaker within the target area, specifically including: S571. Based on the distance between the orientation information and the center of the face in each real-time motion frame, determine the set of candidate speakers and the distance deviation value between the orientation information and each candidate speaker.

[0106] In this embodiment, the sound source location information is combined with preset camera calibration parameters, and a spatial coordinate transformation algorithm is used to map the angle and distance of the sound source in three-dimensional space to the two-dimensional pixel coordinate system of the image, obtaining the theoretical projection coordinate point of the sound source in the current real-time motion picture. Subsequently, the system traverses all faces currently in a stable tracking state and calculates the Euclidean distance between the theoretical projection coordinate point and the center coordinate of each face detection box, thereby quantifying the spatial proximity of each face to the sound source position. Next, the calculated distance value is compared with a preset distance threshold, and targets that are too far away or significantly exceed the effective radiation range of the sound source are eliminated, thereby selecting one or more faces that are close to the sound source and conform to spatial logic as a candidate speaker set. At the same time, the specific distance value between each retained face and the sound source projection point is recorded as the corresponding distance deviation value. This deviation value intuitively reflects the degree of match between the candidate face position and the actual sound source location. The smaller the deviation, the more likely the face is to be the speaker, providing a key quantitative basis for accurately locating the current speaker through comprehensive indicators.

[0107] S572. The distance deviation value and the face tracking confidence of each candidate speaker are weighted and summed to obtain the comprehensive index corresponding to each candidate speaker. The candidate speaker with the largest comprehensive index is the current speaker in the target area.

[0108] In this embodiment, the distance deviation value is normalized and converted into an evaluation factor of the same order of magnitude as the confidence level. Based on a pre-set weighting strategy, corresponding weight coefficients are assigned to the spatial distance factor and the tracking confidence factor. For each target in the candidate speaker set, its comprehensive index is calculated using a weighted summation formula. This involves linearly fusing the normalized distance deviation value with the face tracking confidence level using the weight coefficients to obtain a numerical score that comprehensively reflects the accuracy of the target's spatial location and the reliability of visual tracking. After calculation, the comprehensive indices of all candidate speakers are ranked. The face with the highest comprehensive index value is identified as the speaker who best matches the current sound source location and has the best tracking status, and this face is ultimately locked as the current speaker. This effectively balances the spatial error of sound source localization and the uncertainty of visual tracking, ensuring the accuracy of the speaker recognition result.

[0109] This invention, through the acquisition of sound source information using a microphone array and its localization, analyzes the localization results and face tracking results to determine the current speaker, thus achieving the goal of identifying the current speaker's image. This step utilizes speech activity detection and least squares algorithms to accurately extract the sound source location, not only eliminating interference from environmental noise and random errors and significantly improving the robustness of localization, but also combining visual confidence assessment and motion image analysis. By constructing a comprehensive index model that includes spatial distance deviation and tracking confidence, invalid targets in occluded, blurred, or non-voice-producing areas are accurately eliminated. This audio-visual fusion determination mechanism can accurately locate the current speaker within the target area, effectively solving the problem of misidentification in complex scenarios such as multiple overlapping speakers, mixed interfering sound sources, and rapid target movement, thereby significantly improving the system's accuracy and real-time response capability in speaker identification.

[0110] In one embodiment of the present invention, step 600, based on preset layout rules, generates a multi-person composite image according to the real-time motion images of the current speaker and other people, specifically including: First, the system counts all currently detected participants within the target area and locks the current speaker as the highest priority focus target, with the remaining participants serving as secondary display objects. Then, based on the number of participants and their voice activation status, the system adaptively matches a corresponding layout scheme from a pre-set layout rule base, for example: (1) In a single-person scenario, a full-screen layout is adopted, with the focus image filling the entire output screen, the face displayed in the center, and the screen scaling ratio adaptively adjusted according to the face size to ensure that the face occupies 30%-40% of the screen. (2) In a 2-person scene, a split-screen layout is adopted, with each of the two focused screens occupying 50% of the screen width and full screen height. The faces are displayed in the center and the screen spacing is 10 pixels. (3) In a 3-person scenario, a left-middle-right split-screen layout is adopted, with the screen width divided equally among the three, the height being full screen, and the faces being displayed in the center. The screen spacing is 10 pixels. When voice stimulation is triggered, all split-screen layouts are adopted, with the speaker occupying the full screen on the left or right side, and the remaining two people equally dividing the remaining half of the screen. (4) Two layouts are supported in the 4-person scenario. The default layout is a 2×2 grid layout, with the four focus screens equally divided into screen areas. When voice stimulation is triggered, the layout is switched to "1+3" speaker layout, with the speaker screen occupying 60% of the left area and the other three screens each occupying 13.3% of the right area. (5) In scenes with 5 to 8 people, the grid is three grids at the top and two grids at the bottom when there are 5 people. The upper half of the screen is divided into three equal parts, and the lower half of the screen is centered with the same size. The grid is 2×3 when there are 6 people, and 3×3 when there are 7 to 8 people (8 grids are filled when there are 8 people, and one blank grid is left when there are 7 people). The faces in each focused scene are centered.

[0111] After determining the layout grid, focused cropping is performed on the real-time motion of each face. The face center position, based on the center of the eyes, is precisely calculated using facial coordinates and key facial features. A cropping area is then generated according to the required size proportions of the layout, ensuring the face is centered and meets preset proportion requirements. Furthermore, a dynamic adjustment mechanism is introduced to monitor the face's movement trajectory in real time and update the coordinates of the cropping area synchronously. This ensures that the adjustment speed of the cropped image matches the face's movement speed, effectively preventing image jitter caused by target movement.

[0112] Finally, all cropped focus images are seamlessly stitched and rendered according to the selected layout strategy. When a change in the number of participants or a change in the speaker is detected, the system automatically executes a smooth transition and switch of the layout scheme within 100ms or when the voice stimulus is triggered, thereby generating a multi-person composite image that is clear, well-composed, and highlights the key points.

[0113] In one embodiment of the present invention, after determining the multi-person composite image, it can be output via UVC, HDMI, or media streaming.

[0114] One embodiment of the present invention proposes a multi-person image synthesis device, which can be referred to. Figure 2 , Figure 2 This is a structural block diagram of a multi-person image synthesis device according to an embodiment of the present invention. The device includes: The image acquisition module 201 is used to acquire original images of the target area from multiple angles in real time using a multi-view camera, and preprocess the original images to obtain processed standard images; based on the feature points in all standard images, feature point matching and image stitching are performed to obtain a panoramic image of the current target area.

[0115] The face tracking module 202 is used to perform face detection on the panoramic image based on the preset face recognition technology, determine the position information of all faces in the current frame, and perform real-time tracking and synchronous acquisition of sound source information based on the position information of all faces in the current frame, thereby generating real-time audio signals of the target area and real-time motion images of each face.

[0116] The image synthesis module 203 is used to locate the sound source and stimulate the voice based on the real-time audio signal and real-time motion image to determine the current speaker in the target area; based on the preset layout rules, it generates a multi-person composite image based on the real-time motion image of the current speaker and the real-time motion images of other people.

[0117] The multi-person image synthesis device proposed in this invention generates high-quality panoramic images by real-time acquisition of multi-angle images from a multi-camera array, performing feature point matching and image stitching to achieve wide-area view coverage and obtain more detailed images. Combining face recognition and real-time tracking technology, it simultaneously locks the positions of all faces and collects corresponding audio source information. Utilizing sound source localization and voice stimulation, it intelligently identifies the current speaker and dynamically generates a multi-person composite image that highlights the current speaker while also considering other participants, based on preset layout rules. This enhances the dynamic adaptability of the meeting scenario, automatically focusing on the current speaker and ensuring the clear presentation of key content while maintaining the integrity of panoramic information. It effectively solves the problems of limited viewing angles and information overload in panoramic images associated with traditional devices, improving communication efficiency and viewing experience in remote meetings.

[0118] Figure 3 An internal structural diagram of a computer device according to one embodiment of the present invention is shown. This computer device can specifically be a terminal or a system. Figure 3 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program causes the processor to perform the steps in the above-described method embodiments. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the steps in the above-described method embodiments. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps in the above method embodiments.

[0120] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the steps in the above method embodiments.

[0121] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A method for compositing multiple images, characterized in that, The method includes: A multi-view camera is used to acquire original images of the target area from multiple angles in real time, and the original images are preprocessed to obtain a processed standard image. Feature point matching and image stitching are performed based on feature points in all the standard images to obtain a panoramic image of the current target area. Based on the preset face recognition technology, face detection is performed on the panoramic image to determine the position information of all faces in the current frame; Based on the position information of all faces in the current frame, real-time tracking and synchronous acquisition of sound source information are performed to generate real-time audio signals of the target area and real-time motion images of each face. Based on the real-time audio signal and the real-time motion image, sound source localization and voice stimulation are performed to determine the current speaker in the target area; Based on preset layout rules, a multi-person composite image is generated according to the real-time motion images of the current speaker and other people.

2. The method as described in claim 1, characterized in that, The step of performing feature point matching and image stitching based on feature points in all the standard images to obtain a panoramic image of the current target area specifically includes: Based on the scale-invariant feature transform algorithm, feature points in all the current standard images are detected to obtain feature point information corresponding to each standard image, wherein the target standard image is any one of the standard images; Based on the K-nearest neighbor matching strategy, feature point matching is performed according to the feature point information corresponding to all the standard images to obtain an initial set of feature point matching pairs. Using the random sampling consensus algorithm, mismatched pairs in the initial feature point matching pair set are removed to obtain the remaining target feature point matching pair set. Based on the target feature point matching set and the standard image, the images are stitched together to obtain a panoramic image of the current target area.

3. The method as described in claim 2, characterized in that, The step of stitching together the target feature point matching set and the standard image to obtain a panoramic image of the current target region specifically includes: Based on the target feature point matching, the set and the standard image are spatially aligned to determine the overlapping area of ​​the standard images corresponding to adjacent lenses of the multi-camera after the alignment. Based on the linear brightness adjustment algorithm, the brightness is adjusted according to the brightness of the overlapping area of ​​the standard images corresponding to the adjacent lenses to obtain the adjusted target image; According to the multi-resolution fusion algorithm, the overlapping areas of the target images corresponding to adjacent shots are fused in layers to obtain a panoramic image of the current target area.

4. The method as described in claim 1, characterized in that, The method based on preset face recognition technology performs face detection on the panoramic image to determine the location information of all faces in the current frame, specifically including: The panoramic image is input into a preset YOLOv8 network structure to obtain several candidate face bounding boxes and several candidate face key point sets output by the YOLOv8 network structure. The candidate face bounding boxes include the border position and confidence of the first candidate face, and the candidate face key point sets include the key point positions of the second candidate face. Based on the nonmaximum suppression algorithm, according to the face bounding box position and confidence of each first candidate face, redundant bounding boxes with an overlap higher than a preset overlap threshold are removed from all candidate face bounding boxes to obtain the remaining effective face bounding boxes. Pose estimation is performed based on the facial key point positions of each second candidate face to determine the pose information of the second candidate face; The effective set of facial key points is determined by analyzing the pose information of each second candidate face and the preset angle threshold. Based on all the valid face bounding boxes and the set of valid face key points, determine the position information of all faces in the current frame.

5. The method as described in claim 1, characterized in that, The step of real-time tracking and synchronous acquisition of sound source information based on the position information of all faces in the current frame to generate real-time audio signals for the target area and real-time motion images of each face specifically includes: Using the Kalman filter algorithm, based on the face position information of the target face in the current frame and the obtained state information of the target face, the predicted position region of the target face in the panoramic image of the next frame is predicted, wherein the target face is any one of all faces in the current frame; Based on a preset target tracking algorithm, face tracking is performed in the predicted location area of ​​the target face to determine the real-time position of the target face in the panoramic image of the next frame. Based on the real-time position of the target face, a rectangular image region centered on the face is cropped from the panoramic image of the next frame to generate a real-time motion picture of the target face. The size of the rectangular image region is preset and its area is smaller than the area of ​​the panoramic image. The raw audio signal of the target area is acquired in real time using a preset multi-microphone array; The original audio signal of the target area is denoised and enhanced to generate a preprocessed real-time audio signal.

6. The method as described in claim 1, characterized in that, The step of determining the current speaker within the target area by locating the sound source and stimulating speech based on the real-time audio signal and the real-time motion image specifically includes: Based on the speech activity detection algorithm, noise filtering is performed on the real-time audio signal corresponding to each microphone in the multi-microphone array to obtain a target real-time audio signal containing only valid speech signals. Synchronize and align the target real-time audio signal corresponding to each microphone to determine the start time characteristics of the effective speech signal; A timestamp set is generated based on the timestamp of the target real-time audio signal corresponding to each microphone, according to the start time characteristics of the effective voice signal; Determine the set of time differences between all microphones in the multi-microphone array based on the set of timestamps; The directional information of the effective speech signal is obtained by performing least squares calculation based on the time difference set and the position information of each microphone in the multi-microphone array. Confidence analysis is performed on the real-time motion images of each face to determine the tracking confidence level for each face. Based on the location information, face tracking confidence level, and real-time motion images of each face, the current speaker within the target area is determined.

7. The method as described in claim 6, characterized in that, The step of determining the current speaker within the target area based on the location information, face tracking confidence level, and real-time motion images of each face specifically includes: Based on the distance between the location information and the center of the face in each real-time motion frame, a set of candidate speakers is determined, as well as the distance deviation value between the location information and each candidate speaker; The distance deviation value and the face tracking confidence of each candidate speaker are weighted and summed to obtain a comprehensive index for each candidate speaker. The candidate speaker with the highest comprehensive index is the current speaker in the target area.

8. A multi-person image synthesis device, characterized in that, The device includes: The image acquisition module is used to acquire original images of the target area from multiple angles in real time using a multi-view camera, and preprocess the original images to obtain processed standard images; based on the feature points in all the standard images, feature point matching and image stitching are performed to obtain a panoramic image of the current target area; The face tracking module is used to perform face detection on the panoramic image based on preset face recognition technology, determine the position information of all faces in the current frame, and perform real-time tracking and synchronously collect sound source information based on the position information of all faces in the current frame to generate real-time audio signals of the target area and real-time motion images of each face. The image synthesis module is used to perform sound source localization and voice stimulation based on the real-time audio signal and the real-time motion image, and determine the current speaker in the target area; based on preset layout rules, it generates a multi-person composite image based on the real-time motion image of the current speaker and the real-time motion images of other people.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images