A Multi-User Pose Fusion and Interactive Display System Based on Multi-Channel Image Data

Abstract

This invention discloses a multi-person pose fusion and interactive display system based on multi-channel image data, belonging to the field of computer vision and human-computer interaction technology. It includes: a multi-camera acquisition and analysis module, a data coupling module, a real-time transmission module, and a 3D reconstruction and adaptive correction module. The multi-camera acquisition and analysis module uses a pose estimation engine to obtain the 3D coordinates of skeletal key points and a segmentation and matting module to obtain a target mask. The data coupling module associates user identifiers with multi-view geometry and temporal consistency scores to generate skeletal pose and jitter scores. The real-time transmission module performs backpressure transmission based on queue backlog depth. The 3D reconstruction and adaptive correction module drives virtual characters and occlusion rendering in a 3D rendering engine. This invention solves the problem of identity loss under multi-person occlusion and rapid movement, significantly improving recognition accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction technology, specifically to a multi-person posture fusion and interactive display system based on multi-channel image data. Background Technology

[0002] In the current fields of computer vision and human-computer interaction, large-space multi-person posture capture and virtual interaction applications are becoming increasingly popular. To build a highly immersive virtual reality experience, the system usually needs to collect and process data from multiple visual sensors to recreate the skeletal dynamics of a real person. However, in multi-person, high-dynamic scenes, mutual occlusion between people occurs frequently, making it difficult for a single-view sensor to obtain complete posture information. At the same time, such systems usually involve complex heterogeneous computing architectures, such as artificial intelligence inference and 3D graphics rendering running on different operating systems. The cross-platform communication bandwidth and synchronization mechanism directly affect the end-to-end latency of the system. Traditional system architectures often lack dynamic control and deep integration mechanisms for data stream transmission, which can easily lead to screen stuttering, motion delays, and a decline in the interactive experience.

[0003] While existing technologies employ pose estimation methods based on multi-camera collaboration, these methods often lack robust identity association mechanisms when dealing with multi-source data coupling in complex scenes. When faced with rapid movement or severe occlusion of characters, existing geometric matching algorithms are prone to feature loss or identity recognition abrupt changes, failing to maintain long-term tracking stability. Furthermore, most existing cross-process or cross-device data transmission schemes do not introduce adaptive backpressure adjustment strategies, which can easily lead to data queue backlog and processing delays when the processing rates of the production and consumption ends are mismatched. Meanwhile, in the 3D reconstruction stage, traditional filtering algorithms struggle to achieve a dynamic balance between smooth denoising and high-sensitivity response, and virtual-real occlusion rendering often relies on simple layer overlay, lacking accurate spatial relationship mapping based on depth information, making it difficult to meet the requirements of high-precision interactive display.

[0004] Therefore, how to provide a multi-person pose fusion and interactive display system based on multi-channel image data that can effectively solve the problems of occlusion and unstable identity recognition, achieve low-latency cross-system transmission, and take into account both smooth motion and accurate occlusion rendering is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-person pose fusion and interactive display system based on multi-channel image data. This system addresses the problems of user identification loss or misidentification in complex situations involving multiple users occluding each other or moving rapidly, as well as the high latency and data accumulation issues caused by the lack of flow control in communication between heterogeneous operating systems. Furthermore, it balances the smoothness and responsiveness of skeletal data and achieves pixel-level precise virtual-real depth occlusion effects. Specifically, the technical solution of this invention is as follows:

[0006] The multi-camera acquisition and analysis module is configured to receive image data from multiple RGB cameras, including a motion capture camera group and a keying camera group that operate independently.

[0007] The multi-camera acquisition and analysis module processes or calls the pose estimation engine to process image data to obtain the three-dimensional coordinates of each user's skeletal key points.

[0008] The multi-camera acquisition and analysis module generates or acquires the target mask and two-dimensional bounding box of each frame of the image from the matte camera group.

[0009] The data coupling module is configured to match and associate the 3D coordinates of the skeletal key points output by the motion capture camera group with the target mask output by the keying camera group based on the frame number or timestamp.

[0010] The data coupling module associates the target mask with the corresponding user identifier based on the multi-view geometry and temporal consistency score, generating associated data.

[0011] The associated data includes world coordinate system skeleton pose, two-dimensional bounding box, target mask and skeleton jitter score;

[0012] The real-time transmission module is configured to establish a communication channel based on an asynchronous message communication mechanism, which includes publish-subscribe, push-pull, or a combination thereof. The communication channel supports deployment across operating systems or within the same operating system.

[0013] The real-time transmission module monitors the backlog depth of the current transmission queue.

[0014] The real-time transmission module performs a backpressure transmission strategy on the associated data based on the backpressure depth and transmits the associated data to the three-dimensional reconstruction and adaptive correction module.

[0015] The 3D reconstruction and adaptive correction module is configured to decode associated data in the 3D rendering engine;

[0016] The 3D reconstruction and adaptive correction module drives the virtual character based on the skeletal tremor score.

[0017] The 3D reconstruction and adaptive correction module performs occlusion rendering based on the target mask and outputs the occlusion rendering results to the display module or display interface.

[0018] Preferably, the data coupling module calculates the multi-view geometric and temporal consistency score, including:

[0019] The 3D coordinates of the key points of the skeleton are projected onto the viewport plane corresponding to the chroma key camera group to obtain the skeleton projection coordinates;

[0020] Calculate the spatial overlap between the bone projection coordinates and the target mask and 2D bounding box;

[0021] Calculate the consistency between the bone projection coordinates and the outline of the target mask and 2D bounding box;

[0022] By combining spatial overlap, contour consistency, and preset skeletal keypoint confidence, multi-view geometry and temporal consistency scores are calculated.

[0023] Preferably, the data coupling module associates data based on multi-view geometry and temporal consistency scoring, including:

[0024] When the multi-view geometry and temporal consistency score is not lower than the preset confidence threshold, the association between the target mask and the user identifier is confirmed to be effective, and the association data is generated.

[0025] When the multi-view geometry and temporal consistency score is lower than the preset confidence threshold, the association between the target mask and the user identifier is confirmed to be invalid, and the target mask is invalidated (including: discarding, marking invalid, not participating in occlusion rendering, or using a historical frame mask as a substitute).

[0026] Preferably, the real-time transmission module executes a backpressure transmission strategy, including:

[0027] When the backlog depth is not lower than the preset high water level threshold, congestion control processing is triggered to meet transmission delay constraints.

[0028] When the backlog depth is lower than the preset high water level threshold, normal transmission processing is performed, and the full amount of associated data generated frame by frame is sent with frame sequence features, including frame number or timestamp.

[0029] Preferably, the 3D reconstruction and adaptive correction module drives the virtual character based on the skeletal tremor score, including:

[0030] When the bone jitter score is not lower than the preset jitter threshold, the first smoothing algorithm is used to smooth the world coordinate system bone pose in the associated data and drive the virtual character.

[0031] When the bone jitter score is lower than the preset jitter threshold, the second smoothing algorithm is used to process the world coordinate system bone pose in the associated data and drive the virtual character.

[0032] The first smoothing algorithm has a higher smoothing strength than the second smoothing algorithm.

[0033] Preferably, the 3D reconstruction and adaptive correction module performs occlusion rendering, including:

[0034] Based on the two-dimensional bounding box and target mask in the associated data, construct a foreground rendering layer and a background rendering layer;

[0035] Obtain the depth sorting results of virtual characters and objects in the virtual scene;

[0036] Based on the depth sorting results, occlusion-aware rendering or compositing is performed to correctly reflect the spatial occlusion relationship between virtual characters and objects in the virtual scene during display.

[0037] Preferably, the system also includes:

[0038] The system calibration module is configured to calculate the intrinsic and extrinsic parameter matrices of multiple RGB cameras before the multi-camera acquisition and analysis module runs.

[0039] The system calibration module establishes a unified world coordinate system based on the intrinsic and extrinsic parameter matrices.

[0040] Preferably, the system further includes a display module, which is configured for single-sided display or front and rear dual-sided display; when it is a front and rear dual-sided display, the keying camera group includes at least two keying cameras; when it is a single-sided display, the keying camera group includes one or two keying cameras.

[0041] Preferably, the associated data includes at least frame number, timestamp, user identifier, world coordinate system skeleton pose, and target mask.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. This system calculates multi-view geometric and temporal consistency scores through a data coupling module, and performs a comprehensive evaluation by combining the spatial overlap and contour consistency of the skeleton projection coordinates and the target mask. Based on the scores, an association determination strategy is executed. By filtering out invalid associations below the confidence threshold, the system effectively solves the problem of user identity loss or misidentification in scenarios where multiple people occlude each other or move rapidly, which is common in the background technology. This significantly improves the robustness of multi-path visual data fusion and the accuracy of identity recognition.

[0044] 2. This system establishes a communication channel using a real-time transmission module and implements a backpressure transmission strategy by monitoring the backlog depth of the transmission queue. When the backlog depth is too high, congestion control is triggered to meet transmission delay constraints. When the backlog depth is low, the system sends the full set of associated data generated frame by frame along with frame sequence features, including frame numbers or timestamps. This effectively solves the problem of data accumulation and high end-to-end latency caused by the mismatch in processing rates between heterogeneous systems. While ensuring the continuity of key actions, the system achieves adaptive traffic control, ensuring the real-time nature of the interactive experience.

[0045] 3. This system introduces a skeletal jitter scoring mechanism in the 3D reconstruction and adaptive correction module, realizing adaptive switching of the driving algorithm. By using the first smoothing algorithm to suppress noise when the jitter score is high and the second smoothing algorithm to maintain the response speed when the jitter score is low, the problem of traditional posture driving algorithms being unable to balance motion smoothness and response sensitivity is solved, making the virtual character's movements both stable and able to quickly respond to changes in user posture.

[0046] 4. This system constructs foreground and background rendering layers based on a two-dimensional bounding box and a target mask, and dynamically adjusts the transparency of the foreground rendering layer by combining the depth sorting results of the virtual character and scene objects. It is not limited to transparency; for example, depth buffering or stencil buffering can be used. This mechanism achieves pixel-level depth occlusion rendering, solving the problem of the lack of accurate occlusion processing in the superposition of virtual and real objects in existing technologies. This allows real characters to be correctly presented in front of or behind virtual objects, greatly enhancing the sense of space and immersion in virtual reality interactive scenes. Attached Figure Description

[0047] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0048] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0050] Example 1:

[0051] Please see Figure 1 A multi-person pose fusion and interactive display system based on multi-channel image data, comprising:

[0052] The multi-camera acquisition and analysis module is configured to receive image data from multiple RGB cameras, which include independently operating motion capture camera groups and keying camera groups.

[0053] The multi-camera acquisition and analysis module processes or calls the pose estimation engine to process image data to obtain the three-dimensional coordinates of each user's skeletal key points. The internal implementation of the pose estimation engine is not limited.

[0054] Among them, the multi-camera acquisition and analysis module generates or acquires the target mask and two-dimensional bounding box of each frame of the matting camera group;

[0055] The data coupling module is configured to match and associate the 3D coordinates of the skeletal key points output by the motion capture camera group with the target mask output by the keying camera group based on the frame number or timestamp.

[0056] Among them, the data coupling module associates the target mask with the corresponding user identifier based on the multi-view geometry and temporal consistency score, and generates associated data;

[0057] The associated data includes world coordinate system skeleton pose, 2D bounding box, target mask and skeleton jitter score;

[0058] The real-time transmission module is configured to establish a communication channel based on an asynchronous message communication mechanism, which includes publish-subscribe, push-pull or a combination thereof. The communication channel supports deployment across operating systems or within the same operating system.

[0059] Among them, the real-time transmission module monitors the backlog depth of the current transmission queue;

[0060] Among them, the real-time transmission module performs a back pressure transmission strategy on the associated data according to the back pressure depth and transmits the associated data to the three-dimensional reconstruction and adaptive correction module;

[0061] The 3D reconstruction and adaptive correction module is configured to decode associated data in the 3D rendering engine;

[0062] Among them, the 3D reconstruction and adaptive correction module drives the virtual character based on the skeletal tremor score;

[0063] The 3D reconstruction and adaptive correction module performs occlusion rendering based on the target mask and outputs the occlusion rendering results to the display module or display interface.

[0064] This embodiment discloses a multi-person posture fusion and interactive display system based on multi-channel image data; the system aims to solve the problems of single visual sensor being easily occluded, unstable identity recognition, and high transmission latency across operating systems in large-space, multi-person, high-dynamic scenes.

[0065] The core architecture of this system includes a multi-camera acquisition and analysis module, a data coupling module, a real-time transmission module, and a 3D reconstruction and adaptive correction module.

[0066] Multi-camera acquisition and analysis module: This module is configured to receive image data from multiple RGB cameras; in this embodiment, the system deploys... High-speed RGB camera, recommended configuration Within the motion capture camera group and the keying camera group, hardware-triggered synchronization or software-time synchronization can be used, or they can work asynchronously and be aligned by timestamps; they are distributed at a high position in the interactive space to ensure that there are no blind spots in the coverage area;

[0067] Skeletal Extraction: This module integrates a pose estimation engine, which can be a pose estimation model based on convolutional neural networks, but is not limited to this. It processes or calls the pose estimation engine to process image data to obtain the 3D coordinates of each user's skeletal key points. The coordinate set is obtained by decoding the output tensor of the model inference layer, which contains the position information of the key joints of the human body in the local coordinate system of the camera and the two-dimensional bounding box; the image of the keying camera group is called, and the target mask / two-dimensional bounding box is generated inside the analysis module. The target mask is a pixel-level binarized image or alpha channel map, which is used to accurately describe the human body contour.

[0068] Data Coupling Module: This module is the hub connecting visual recognition and 3D interaction. It is configured to match and associate the 3D coordinates of the skeletal key points output by the motion capture camera group with the target mask output by the keying camera group based on the frame number or timestamp.

[0069] Association Mechanism: Based on multi-view geometric and temporal consistency scoring, this module associates the target mask at the 2D level with the corresponding user identifier at the 3D level to generate associated data;

[0070] Associated data structure: generated associated data packets Includes: world coordinate system skeleton pose, 2D bounding box, target mask, and skeleton jitter score; the number of display screens and keying cameras can be freely combined, including single-sided display or front and rear double-sided display, with a corresponding keying camera of at least one;

[0071] Real-time transmission module: This module is configured to establish a communication channel based on an asynchronous message communication mechanism, including publish-subscribe, push-pull, or a combination thereof. The communication channel supports cross-operating system or same-operating system deployment. Since AI computing usually runs in a Linux environment, while 3D rendering mostly runs in a Windows environment, such as 3D rendering engines, this module solves the problem of high-throughput and low-latency transmission across operating systems.

[0072] Back pressure monitoring: This module monitors the back pressure depth of the current transmission queue in real time, that is, the number of data frames waiting to be consumed in the buffer;

[0073] Transmission strategy: Based on the back pressure transmission strategy, the associated data is transmitted to the 3D reconstruction and adaptive correction module to ensure that the receiving end does not experience processing delays due to data accumulation;

[0074] 3D Reconstruction and Adaptive Correction Module: This module is configured to decode associated data in the 3D rendering engine;

[0075] Driving logic: The module drives the virtual character based on the skeletal jitter score, and uses a dynamic filtering algorithm to balance the smoothness of the movements and the response speed;

[0076] Occlusion rendering: The module performs occlusion rendering based on the target mask, mapping the outline of the real-world person to the virtual 3D space in real time, and realizing the correct front-to-back occlusion relationship between virtual objects and real people;

[0077] Through the collaborative work of the above modules, this embodiment realizes an adaptive black-box motion capture + AI keying + cross-platform transmission + 3D rendering system. Its significant effects are: by decoupling multiple modules and data fusion based on scoring, the problem of user identification (ID) loss due to character occlusion in traditional monocular solutions is solved; through the backpressure transmission strategy, the end-to-end latency of cross-operating system communication is effectively reduced. For example, in one embodiment, the 95th percentile latency can be stably controlled within 50ms, which significantly improves the immersion and smoothness of multi-person interactive entertainment.

[0078] Example 2:

[0079] The data coupling module calculates multi-view geometric and temporal consistency scores, including:

[0080] The 3D coordinates of the key points of the skeleton are projected onto the viewport plane corresponding to the chroma key camera group to obtain the skeleton projection coordinates;

[0081] Calculate the spatial overlap between the bone projection coordinates and the target mask and 2D bounding box;

[0082] Calculate the consistency between the bone projection coordinates and the outline of the target mask and 2D bounding box;

[0083] By combining spatial overlap, contour consistency and preset skeletal keypoint confidence, multi-view geometric and temporal consistency scores are calculated.

[0084] This embodiment further illustrates the specific process of the data coupling module in calculating the multi-view geometric and temporal consistency score; the score aims to quantify the matching reliability between 2D image features and 3D skeleton data.

[0085] To accurately assess the degree of matching and eliminate logical errors caused by directly adding indices with different dimensions, this embodiment introduces a normalized comprehensive consistency score. The calculation logic is as follows:

[0086] Projection transformation and spatial overlap calculation:

[0087] The 3D coordinates of the key points of the skeleton are projected onto the viewport plane corresponding to the chroma key camera group to obtain the skeleton projection coordinates;

[0088] Let the first The world coordinate system skeleton point set of each user is superscript A unique identifier for the user, index The world coordinate system is represented by the camera's extrinsic parameter matrix. The intrinsic parameter matrix is Then the projected pixel coordinates The calculation formula is: ;

[0089] in: Scale factor; These are the pixel coordinates projected onto the image plane, acquired by the vision sensor and calculated using the matrix transformation described above.

[0090] The coordinates of the skeleton points in the world coordinate system are derived from the 3D reconstruction output of the multi-camera acquisition and analysis module.

[0091] Calculate the spatial overlap between the bone projection coordinates and the target mask and 2D bounding box; this embodiment uses the intersection-union ratio (IUGR) as the main metric, and also calculates the proportion of projection points falling within the mask area. ;

[0092] Contour consistency calculation: Calculate the contour consistency between the bone projection coordinates and the target mask and 2D bounding box; this is achieved by analyzing the fit between the bone connections and the mask edges;

[0093] The scoring fusion model combines the above indicators to calculate the multi-view geometric and temporal consistency score; to ensure the rigor of the physical meaning of the formula, a normalization function is introduced. The temporal deviation of the distance dimension is transformed into a dimensionless probability space, and the scoring formula is defined as follows: ;

[0094] in: Multi-view geometric and temporal consistency score, value range ; : The intersection-union ratio of the skeletal bounding rectangle and the visual detection 2D bounding box, dimensionless, derived from spatial geometric calculations; The proportion of skeletal key points falling within the target mask; dimensionless, derived from pixel-level statistics. The preset confidence level of the skeletal key points, for example, comes from the average peak probability of the heatmap output by the upstream pose estimation model. The internal implementation of the pose estimation engine and the source of the confidence level are not limited. Temporal position deviation, i.e., temporal consistency deviation based on historical frames, such as the Euclidean distance between the observed position in the current frame and the predicted position of the prediction model, in pixels; the prediction model may use a Kalman filter; The normalized penalty function is defined as follows: ;

[0095] in, The value of is related to the camera image resolution and is usually set to the length of the image diagonal pixels. For example, in full HD At resolution, take Pixel;

[0096] Weighting coefficients, satisfying ; Specific meanings and recommended values ​​are: Corresponding to the overlap weight of the two-dimensional bounding box; Weights corresponding to the distribution of points inside the mask; Corresponding skeletal confidence weights; Corresponding to the temporal deviation weight; the above weight values ​​can be determined by a data-driven method or are configurable; as a preferred embodiment, the above weight values ​​can be determined based on a sample set containing a specific number of frame annotation data (e.g., 10,000 frames), by fitting a logistic regression model with the objective of maximizing the association accuracy. The recommended optimal values ​​for each weight coefficient are as follows: , , , ;

[0097] By introducing a comprehensive scoring mechanism that includes projection geometry and temporal features, and correcting the problem of inconsistent dimensions through a normalization function, this embodiment can greatly improve the robustness of pairing 2D images with 3D data. Even when multiple people overlap or move rapidly, causing a certain indicator to temporarily fail, the system can still maintain the correct ID association based on the distribution ratio of bone points in the target mask or temporal prediction, thus solving the problem of identity jump in complex scenes.

[0098] Example 3:

[0099] The data coupling module associates data based on multi-view geometric and temporal consistency scores, including:

[0100] When the multi-view geometry and temporal consistency score is not lower than the preset confidence threshold, the association between the target mask and the user identifier is confirmed to be effective, and the association data is generated.

[0101] When the multi-view geometry and temporal consistency score is lower than the preset confidence threshold, the association between the target mask and the user identifier is confirmed to be invalid, and the target mask is invalidated (including: discarding, marking invalid, not participating in occlusion rendering, or using a historical frame mask as a substitute).

[0102] This embodiment details the specific logic of the data coupling module in associating data based on multi-view geometry and temporal consistency scores, and uses a state machine strategy to manage the effectiveness of the association.

[0103] Effective correlation: When multi-view geometric and temporal consistency scores Not lower than the preset confidence threshold At that time, the system confirms that the association between the target mask and the user identifier is valid and generates association data; this threshold The threshold value was selected based on receiver operating characteristic (ROC) curve analysis, ensuring an extremely low false alarm rate. In this embodiment... The typical value for is 0.85;

[0104] Invalid associations and discarding: when scoring Below the preset confidence threshold When the system determines that the current visual features are seriously inconsistent with the skeletal model, confirms that the association between the target mask and the user identifier is invalid, and performs invalidation processing on the target mask (including: discarding, marking it as invalid, not participating in occlusion rendering, or using a historical frame mask as a substitute).

[0105] This embodiment effectively filters out low-quality visual feature matches by setting a reasonable confidence threshold, ensuring the continuity of the interactive experience and visual stability.

[0106] Example 4:

[0107] The real-time transmission module executes a backpressure transmission strategy, including:

[0108] When the backlog depth is not lower than the preset high water level threshold, congestion control processing is triggered to meet transmission delay constraints.

[0109] When the backlog depth is lower than the preset high water level threshold, normal transmission processing is performed, and the full associated data generated frame by frame is sent with frame sequence features. The frame sequence features include frame number or timestamp. The full associated data is transmitted with low latency using high-speed network interfaces such as gigabit network cards.

[0110] This embodiment details the backpressure transmission strategy executed by the real-time transmission module; this strategy aims to solve the latency accumulation problem caused by the mismatch between the processing rates of the production end and the consumption end.

[0111] Backlog depth monitoring: The real-time transmission module maintains a transmission queue and reads the backlog depth in real time. ;

[0112] Congestion control processing: When Not lower than the preset high water level threshold When the threshold value reaches a certain threshold, it means that the rendering end's processing capacity is saturated, at which point congestion control is triggered to meet transmission delay constraints; The setting is based on: ;

[0113] in, The maximum queuing delay allowed by the system. This represents the average rendering time per frame.

[0114] Congestion control can be achieved by limiting the transmission rate or by downsampling and compressing the target mask data.

[0115] Normal transmission processing: When Below During normal transmission, the full associated data generated frame by frame is sent along with frame sequence features. The frame sequence features include frame number or timestamp to ensure the most refined presentation of the action. The frame sequence features can be frame number or timestamp.

[0116] This backpressure transmission strategy enables the system to have adaptive flow control capabilities. When there are network fluctuations or excessive load on the rendering end, congestion control is triggered to maintain low system latency, prevent the latency avalanche effect caused by data accumulation, and ensure real-time synchronization between user actions and screen feedback.

[0117] Example 5:

[0118] The 3D reconstruction and adaptive correction module drives the virtual character based on skeletal jitter scores, including:

[0119] When the bone jitter score is not lower than the preset jitter threshold, the first smoothing algorithm is used to smooth the world coordinate system bone pose in the associated data and drive the virtual character.

[0120] When the bone jitter score is lower than the preset jitter threshold, the second smoothing algorithm is used to process the world coordinate system bone pose in the associated data and drive the virtual character.

[0121] It should be noted that the first smoothing algorithm and the second smoothing algorithm in this embodiment can be two different filtering algorithms, such as Kalman filtering and first-order low-pass filtering, or they can be two processing states of the same underlying filtering algorithm under different smoothing parameters, that is, they can be achieved by dynamically adjusting the smoothing parameters or intensity. The specific implementation method is not limited.

[0122] This embodiment details the specific algorithm selection for the 3D reconstruction and adaptive correction module, which drives the virtual character selection based on the skeletal tremor score.

[0123] Skeletal tremor score definition: Skeletal tremor score Defined as the variance of acceleration at skeletal keypoints within a time window, the calculation formula is as follows: ;

[0124] in, This represents the total number of skeletal key points. For the first A key point recently The sequence of acceleration magnitudes within a frame, Indicates variance operation; It can also be defined by velocity, joint angle change, confidence fluctuation, etc. The above are just examples;

[0125] Strong smoothing: when Not lower than the preset jitter threshold When this occurs, it indicates that the original data contains significant noise; at this point, the first smoothing algorithm is used to smooth the world coordinate system skeleton pose in the associated data; the state vector of the Kalman filter... Defined as a six-dimensional vector containing position and velocity: ;

[0126] in The three-dimensional coordinates of the bone points. These are the corresponding three-axis velocity components;

[0127] Kalman filtering uses a prediction-update step to predict the current state using the best estimate from the previous time step, which can effectively suppress high-frequency noise.

[0128] Weak smoothing: when Below When the data quality is good, a first-order low-pass filter algorithm is used; the formula is: ;

[0129] in: : The output pose after filtering; The observed pose of the current frame is derived from the data unpacked by the real-time transmission module; : Output pose of the previous frame; Smoothing factor, ranging from 0.5 to 0.9, is dynamically adjusted based on the current frame rate to ensure a constant cutoff frequency. The specific dynamic adjustment formula is as follows: ;

[0130] Alternatively, a more precise formula can be derived using the time constant form: ;

[0131] in, As the baseline smoothing factor, The current actual frame rate monitored by the real-time transmission module. The system target frame rate; For the current frame interval, The preset time constant;

[0132] This embodiment resolves the contradiction between latency caused by filtering and jitter caused by not filtering by using a dynamic filtering strategy based on jitter scoring. It pursues low latency and high sensitivity when the data is stable, and prioritizes smooth image when the data is unstable, thus providing the best visual experience under different working conditions.

[0133] Example 6:

[0134] The 3D reconstruction and adaptive correction module performs occlusion rendering, including:

[0135] Based on the two-dimensional bounding box and target mask in the associated data, construct a foreground rendering layer and a background rendering layer;

[0136] Obtain the depth sorting results of virtual characters and objects in the virtual scene;

[0137] Based on the depth sorting results, occlusion-aware rendering or compositing is performed to correctly reflect the spatial occlusion relationship between the virtual character and objects in the virtual scene during display: when the depth sorting results indicate that the virtual character is behind an object in the virtual scene, the opacity of the foreground rendering layer is increased to display the object in the virtual scene; when the depth sorting results indicate that the virtual character is in front of an object in the virtual scene, the opacity of the foreground rendering layer is maintained to occlude the object in the virtual scene.

[0138] This embodiment details the mechanism by which the 3D reconstruction and adaptive correction module performs occlusion rendering, which is a key step in achieving virtual-real fusion.

[0139] Layer construction: Based on the 2D bounding box and target mask in the associated data, a dynamic foreground rendering layer and background rendering layer are constructed in the rendering engine; the target mask is mapped onto a transparent bulletin board or grid, which moves with the bone position;

[0140] Depth Sorting and Transparency Control: Depth Acquisition: Real-time acquisition of the depth sorting results of the virtual character and objects in the virtual scene from the camera's perspective; Displaying Virtual Objects: When the depth sorting result indicates that the virtual character is behind an object in the virtual scene, i.e. The system disables depth writing to the foreground rendering layer or sets it to full transparency, allowing virtual objects in the foreground to be clearly displayed; occluding virtual objects: when the depth sorting result indicates that the virtual character is in front of objects in the virtual scene, i.e. The depth test and writing of the foreground rendering layer are enabled, and only the depth information of the target mask area is rendered. At this time, the target mask image of the real person is overlaid on the virtual object, and the visual effect is that the person occludes the virtual object. The above depth sorting and transparency control are only examples of implementation methods. This invention can also realize the spatial front and back occlusion relationship through depth buffering, template buffering or other occlusion perception synthesis methods, which does not constitute a limitation on the specific implementation details.

[0141] This embodiment utilizes 2D keying data to participate in 3D depth testing, achieving pixel-level precise occlusion relationships. Compared to simply overlaying human images on top, this solution allows virtual objects to pass through or bypass real people, greatly enhancing the sense of space and realism in augmented reality interactions.

[0142] Example 7:

[0143] The system also includes: a system calibration module, configured to calculate the intrinsic and extrinsic parameter matrices of multiple RGB cameras before the multi-camera acquisition and analysis module runs;

[0144] The system calibration module establishes a unified world coordinate system based on the intrinsic and extrinsic parameter matrices.

[0145] This embodiment involves a system calibration module; this module is configured to calculate the intrinsic and extrinsic parameter matrices of multiple RGB cameras before the multi-camera acquisition and analysis module runs;

[0146] Calibration method: Using a calibration board or checkerboard with specific coding features, the calibration rod is moved in space, and the algorithm and bundle adjustment solution parameters are determined by perspective pose determination;

[0147] Coordinate System 1: A unified world coordinate system is established based on the intrinsic and extrinsic parameter matrices; all skeleton coordinates and bounding box positions are uniformly transformed to this coordinate system to ensure that data from different camera perspectives are strictly aligned in space.

[0148] Precise system calibration is the foundation of multi-camera fusion; by establishing a unified world coordinate system, parallax errors between different cameras are eliminated, ensuring the geometric accuracy of multi-view data fusion, thereby guaranteeing the spatial accuracy of subsequent skeleton extraction and 3D reconstruction.

[0149] Example 8:

[0150] The system also includes a display module, which is configured for single-sided display or front and rear dual-sided display; when it is a front and rear dual-sided display, the keying camera group includes at least two keying cameras; when it is a single-sided display, the keying camera group includes one or two keying cameras.

[0151] The associated data includes at least the frame number, timestamp, user identifier, world coordinate system skeleton pose, and target mask.

[0152] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-person pose fusion and interactive display system based on multi-path image data, characterized in that, include: The multi-camera acquisition and analysis module is configured to receive image data from multiple RGB cameras, including a motion capture camera group and a keying camera group that operate independently. The multi-camera acquisition and analysis module processes or calls the pose estimation engine to process image data to obtain the three-dimensional coordinates of each user's skeletal key points. The multi-camera acquisition and analysis module generates or acquires the target mask and two-dimensional bounding box of each frame of the image from the matte camera group. The data coupling module is configured to match and associate the 3D coordinates of the skeletal key points output by the motion capture camera group with the target mask output by the keying camera group based on the frame number or timestamp. The data coupling module associates the target mask with the corresponding user identifier based on the multi-view geometry and temporal consistency score, generating associated data. The associated data includes world coordinate system skeleton pose, two-dimensional bounding box, target mask and skeleton jitter score; The real-time transmission module is configured to establish a communication channel based on an asynchronous message communication mechanism, which includes publish-subscribe, push-pull, or a combination thereof. The communication channel supports deployment across operating systems or within the same operating system. The real-time transmission module monitors the backlog depth of the current transmission queue. The real-time transmission module performs a backpressure transmission strategy on the associated data based on the backpressure depth and transmits the associated data to the three-dimensional reconstruction and adaptive correction module. The 3D reconstruction and adaptive correction module is configured to decode associated data in the 3D rendering engine; The 3D reconstruction and adaptive correction module drives the virtual character based on the skeletal tremor score. The 3D reconstruction and adaptive correction module performs occlusion rendering based on the target mask and outputs the occlusion rendering result to the display module or display interface. The 3D reconstruction and adaptive correction module drives the virtual character based on skeletal tremor scores, including: When the bone jitter score is not lower than the preset jitter threshold, the first smoothing algorithm is used to smooth the world coordinate system bone pose in the associated data and drive the virtual character. When the bone jitter score is lower than the preset jitter threshold, the second smoothing algorithm is used to process the world coordinate system bone pose in the associated data and drive the virtual character. The first smoothing algorithm has a higher smoothing strength than the second smoothing algorithm.

2. The multi-person pose fusion and interactive display system based on multi-channel image data according to claim 1, characterized in that, The data coupling module calculates the multi-view geometric and temporal consistency score, including: The 3D coordinates of the key points of the skeleton are projected onto the viewport plane corresponding to the chroma key camera group to obtain the skeleton projection coordinates; Calculate the spatial overlap between the bone projection coordinates and the target mask and 2D bounding box; Calculate the consistency between the bone projection coordinates and the outline of the target mask and 2D bounding box; By combining spatial overlap, contour consistency, and preset skeletal keypoint confidence, multi-view geometry and temporal consistency scores are calculated.

3. The multi-person pose fusion and interactive display system based on multi-channel image data according to claim 2, characterized in that, The data coupling module associates data based on multi-view geometry and temporal consistency scores, including: When the multi-view geometry and temporal consistency score is not lower than the preset confidence threshold, the association between the target mask and the user identifier is confirmed to be effective, and the association data is generated. When the multi-view geometry and temporal consistency score is lower than the preset confidence threshold, the association between the target mask and the user identifier is confirmed to be invalid, and the target mask is invalidated.

4. The multi-person pose fusion and interactive display system based on multi-channel image data according to claim 1, characterized in that, The real-time transmission module executes a backpressure transmission strategy, including: When the backlog depth is not lower than the preset high water level threshold, congestion control processing is triggered to meet transmission delay constraints. When the backlog depth is lower than the preset high water level threshold, normal transmission processing is performed, and the full amount of associated data generated frame by frame is sent with frame sequence features, including frame number or timestamp.

5. A multi-person pose fusion and interactive display system based on multi-channel image data according to claim 1, characterized in that, The 3D reconstruction and adaptive correction module performs occlusion rendering, including: Based on the two-dimensional bounding box and target mask in the associated data, construct a foreground rendering layer and a background rendering layer; Obtain the depth sorting results of virtual characters and objects in the virtual scene; Based on the depth sorting results, occlusion-aware rendering or compositing is performed to correctly reflect the spatial occlusion relationship between virtual characters and objects in the virtual scene during display.

6. A multi-person pose fusion and interactive display system based on multi-channel image data according to claim 1, characterized in that, Also includes: The system calibration module is configured to calculate the intrinsic and extrinsic parameter matrices of multiple RGB cameras before the multi-camera acquisition and analysis module runs. The system calibration module establishes a unified world coordinate system based on the intrinsic and extrinsic parameter matrices.

7. A multi-person pose fusion and interactive display system based on multi-channel image data according to claim 1, characterized in that, It also includes a display module, which is configured for single-sided display or front and rear dual-sided display; when it is a front and rear dual-sided display, the keying camera group includes at least two keying cameras; when it is a single-sided display, the keying camera group includes one or two keying cameras.

8. A multi-person pose fusion and interactive display system based on multi-channel image data according to claim 1, characterized in that, The associated data includes at least the frame number, timestamp, user identifier, world coordinate system skeleton pose, and target mask.

Images