Hybrid rendering method and apparatus

Abstract

The application provides a mixed rendering method and device, constructs a multi-dimensional semantic depth fusion field including a local skeleton semantic field, a high-precision relative depth field, a dynamic timing motion field and an edge light reflection field based on a real person video stream, determines a per-pixel occlusion decision based on the local skeleton semantic field and the high-precision relative depth field, combines skeleton semantics and depth information simultaneously in the occlusion relationship, reduces false occlusion and occlusion state jump under the condition of fast motion and limb interlacing, determines a collision avoidance vector based on the dynamic timing motion field, enables a virtual element to adjust a position according to timing motion information of the real person, determines a transfer light mapping based on the edge light reflection field, enables the real person edge light to form a continuous transition with virtual scene light, and determines a target Alpha mask based on the local skeleton semantic field, the high-precision relative depth field and the dynamic timing motion field, and reduces flicker and trailing of virtual-real edges.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a hybrid rendering method and apparatus. Background Technology

[0002] In a mixed-real performance scene, it is necessary to perform real-time mixed rendering of live video streams with virtual dancers, virtual props, and stage elements.

[0003] Existing technologies typically determine occlusion relationships by using a fixed layer order or by comparing real-person depth information with virtual scene depth information pixel by pixel.

[0004] When real human limbs intersect with virtual elements or their spatial relationships dynamically change, a fixed layer order cannot adjust the occlusion order according to changes in spatial relationships. Relying solely on depth information is easily affected by depth noise and edge blurring caused by rapid limb movements, leading to clipping, incorrect occlusion, or abrupt changes in occlusion state. At the same time, existing technologies typically adjust the position of virtual elements through manual arrangement or geometric detection after collisions, making it difficult to pre-determine collision avoidance methods using the temporal motion information of real humans. During human image separation and virtual scene rendering, there is a lack of correlation between the lighting information of real human edges and the lighting parameters of the virtual engine, which can easily cause abrupt changes in lighting at the virtual and real edges. The lack of correlation between skeletal semantics, depth, and motion information between alpha masks generated frame by frame can easily produce edge flickering or ghosting. Summary of the Invention

[0005] In view of this, the present invention provides a hybrid rendering method and apparatus, which effectively improves the stability and continuity of hybrid rendering of virtual and real elements when real people interact closely with virtual elements.

[0006] The first aspect of this invention provides a hybrid rendering method, comprising:

[0007] Acquire live video streams and virtual scene information;

[0008] Based on the live video stream, a multidimensional semantic deep fusion field is constructed; wherein, the multidimensional semantic deep fusion field includes a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge illumination reflection field;

[0009] Based on the local skeletal semantic field, a pixel-by-pixel occlusion decision is determined;

[0010] Based on the dynamic temporal motion field, a collision avoidance vector is determined;

[0011] Based on the edge illumination reflection field and the illumination parameters of the virtual engine, the transfer illumination map is determined;

[0012] Based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field, the target alpha mask is determined.

[0013] Based on the pixel-by-pixel occlusion decision, the collision avoidance vector, the transitive lighting mapping, and the target alpha mask, the live video stream and the virtual scene are rendered using a hybrid rendering method.

[0014] Optionally, constructing a multi-dimensional semantic deep fusion field based on the live video stream includes:

[0015] Obtain live video stream;

[0016] Human pose estimation is performed based on the live video stream to obtain the human pose estimation result, and human figure segmentation is performed on the live subject in the live video stream to obtain the human figure segmentation mask.

[0017] Based on the human pose estimation results, a local skeletal semantic field is constructed;

[0018] Based on the initial depth map corresponding to the live video stream and the local skeletal semantic field, a high-precision relative depth field is constructed;

[0019] Based on the human pose estimation results of consecutive frames, a dynamic temporal motion field is constructed;

[0020] Based on the portrait segmentation mask and the live video stream, an edge illumination reflection field is constructed;

[0021] A multidimensional semantic depth fusion field is constructed based on the local skeletal semantic field, the high-precision relative depth field, the dynamic temporal motion field, and the edge illumination reflection field.

[0022] Optionally, the human pose estimation result includes the spatial location of skeletal joints, and the construction of a local skeletal semantic field based on the human pose estimation result includes:

[0023] Determine the skeletal axis based on the spatial location of skeletal joints;

[0024] Determine the projection position of the spatial location to be processed on each bone axis;

[0025] Based on the projection position and the spatial position to be processed, determine the semantic attribution strength of the spatial position to be processed for each bone segment;

[0026] Based on the semantic attribution strength of each bone segment to the spatial location to be processed, the semantic segment label of the bone corresponding to the spatial location to be processed is determined.

[0027] Based on the skeletal semantic segment tags corresponding to each of the spatial locations to be processed, a local skeletal semantic field is constructed.

[0028] Optionally, constructing a high-precision relative depth field based on the initial depth map corresponding to the live-action video stream and the local skeletal semantic field includes:

[0029] Based on the bone semantic segment markers in the local bone semantic field, the pixels in the initial depth map are divided into pixel sets corresponding to different bone segments;

[0030] Based on the pixel set corresponding to the bone segment and the initial depth map, the target depth value of each pixel of the bone segment is determined;

[0031] The target depth value of the adjacent boundary pixels is determined based on the adjacent boundary pixels between adjacent bone segments and a preset depth gradient threshold.

[0032] A high-precision relative depth field is constructed based on the target depth values ​​of each pixel in the bone segment and the target depth values ​​of the adjacent boundary pixels.

[0033] Optionally, constructing the edge illumination reflection field based on the portrait segmentation mask and the live video stream includes:

[0034] Based on the boundary of the portrait segmentation mask and the preset edge band width, the set of edge pixels of the portrait segmentation mask is determined;

[0035] The ambient light component is determined based on the brightness of the neighboring pixels of the edge pixels in the edge pixel set of the portrait segmentation mask.

[0036] Based on the edge gradient of the portrait segmentation mask, the edge normal vector corresponding to the edge pixel is determined;

[0037] Based on the edge normal vector and the direction of the main light source in the ambient lighting information, the directional light component is determined;

[0038] Based on the color channel data of the edge pixels, the reflected light components are determined;

[0039] The edge illumination reflection field is constructed based on the ambient light component, directional light component, and reflected light component corresponding to each edge pixel.

[0040] Optionally, determining the pixel-by-pixel occlusion decision based on the local skeletal semantic field and the high-precision relative depth field includes:

[0041] Query the skeletal semantic segment marker of the target pixel from the local skeletal semantic field;

[0042] Based on the skeletal semantic segment marker of the target pixel, the target bone segment to which the target pixel belongs is determined;

[0043] The target depth values ​​of all pixels within the target bone segment are obtained from the high-precision relative depth field, and the scene depth value of the virtual scene at the target pixel position is obtained.

[0044] Based on the target depth values ​​of all pixels within the target bone segment, determine the segment-level aggregation depth;

[0045] Based on the segment-level aggregation depth and the scene depth value, the occlusion decision of the target pixel is determined.

[0046] Optionally, the virtual scene information includes: virtual element position information; determining the collision avoidance vector based on the dynamic temporal motion field includes:

[0047] Based on the virtual element position information, determine the virtual element vertices and the projection position of the virtual element vertices in screen space.

[0048] The predicted displacement corresponding to the projection position is determined based on the dynamic temporal motion field.

[0049] Based on the predicted displacement, preset safety threshold, and offset coefficient corresponding to the projected position, a collision avoidance vector is determined.

[0050] Optionally, the virtual scene information includes: lighting parameters of the virtual engine; determining the transfer lighting map based on the edge lighting reflection field includes:

[0051] Extract the ambient light component, directional light component, and reflected light component of the edge pixels from the edge illumination reflection field;

[0052] Query the original lighting parameters corresponding to the spatial position of the edge pixel in the lighting parameters of the virtual engine;

[0053] Based on the edge pixels and the portrait segmentation mask, determine the spatial attenuation weight;

[0054] Based on the ambient light component, the directional light component, the reflected light component, the original illumination parameters, and the spatial attenuation weight, the transmitted illumination map is determined.

[0055] Optionally, determining the target alpha mask based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field includes:

[0056] Based on the local skeletal semantic field and the dynamic temporal motion field, the target neighboring pixels are determined from the neighboring pixels of the target pixel;

[0057] Based on the target pixel and the target neighboring pixels, determine the spatial smoothness constraint;

[0058] Based on the high-precision relative depth field, the target pixel, and the neighboring pixels of the target pixel, depth continuity constraints are determined;

[0059] Based on the dynamic temporal motion field and the target pixel position in the current frame, determine the corresponding pixel position of the target pixel in the previous frame;

[0060] Based on the current frame alpha value of the target pixel and the previous frame alpha value of the corresponding pixel position, a temporal consistency constraint is determined.

[0061] The target alpha mask is determined based on the spatial smoothness constraint, the depth continuity constraint, and the temporal consistency constraint.

[0062] A second aspect of the present invention provides a hybrid rendering apparatus, comprising:

[0063] The acquisition unit is used to acquire live video streams and virtual scene information;

[0064] A multidimensional semantic deep fusion field construction unit is used to construct a multidimensional semantic deep fusion field based on the real-person video stream; wherein, the multidimensional semantic deep fusion field includes a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge illumination reflection field;

[0065] A pixel-by-pixel occlusion decision determination unit is used to determine a pixel-by-pixel occlusion decision based on the local skeletal semantic field and the high-precision relative depth field.

[0066] The collision avoidance vector determination unit is used to determine the collision avoidance vector based on the dynamic temporal motion field.

[0067] A transmission illumination mapping determination unit is used to determine the transmission illumination mapping based on the edge illumination reflection field;

[0068] The target alpha mask determination unit is used to determine the target alpha mask based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field.

[0069] The virtual-real hybrid rendering unit is used to perform virtual-real hybrid rendering on the live video stream and the virtual scene based on the pixel-by-pixel occlusion decision, the collision avoidance vector, the transitive lighting map, and the target alpha mask.

[0070] Optionally, the multidimensional semantic deep fusion field construction unit includes:

[0071] The live video stream acquisition unit is used to acquire live video streams.

[0072] The video stream processing unit is used to perform human pose estimation based on the live video stream, obtain human pose estimation results, and perform human image segmentation on the live subject in the live video stream to obtain human image segmentation mask.

[0073] A local skeletal semantic field construction unit is used to construct a local skeletal semantic field based on the human pose estimation results.

[0074] A high-precision relative depth field construction unit is used to construct a high-precision relative depth field based on the initial depth map corresponding to the real-person video stream and the local skeletal semantic field.

[0075] The dynamic temporal motion field construction unit is used to construct a dynamic temporal motion field based on the human pose estimation results of consecutive frames.

[0076] An edge illumination reflection field construction unit is used to construct an edge illumination reflection field based on the portrait segmentation mask and the live video stream;

[0077] A multidimensional semantic deep fusion field construction subunit is used to construct a multidimensional semantic deep fusion field based on the local skeletal semantic field, the high-precision relative depth field, the dynamic temporal motion field, and the edge illumination reflection field.

[0078] Optionally, the local skeletal semantic field construction unit includes:

[0079] The skeletal axis determination unit is used to determine the skeletal axis based on the spatial position of skeletal joints.

[0080] The projection position determination unit is used to determine the projection position of the spatial position to be processed on each bone axis;

[0081] A semantic attribution strength determination unit is used to determine the semantic attribution strength of the spatial position to be processed for each bone segment based on the projection position and the spatial position to be processed.

[0082] The skeletal semantic segment labeling determination unit is used to determine the skeletal semantic segment label corresponding to the spatial location to be processed based on the semantic attribution strength of each skeletal segment to the spatial location to be processed.

[0083] The local skeletal semantic field construction subunit is used to construct a local skeletal semantic field based on the skeletal semantic segment tags corresponding to each of the spatial locations to be processed.

[0084] Optionally, the high-precision relative depth field construction unit includes:

[0085] The partitioning unit is used to divide the pixels in the initial depth map into sets of pixels corresponding to different bone segments based on the bone semantic segment labels in the local bone semantic field.

[0086] The first target depth value determination unit is used to determine the target depth value of each pixel of the bone segment based on the pixel set corresponding to the bone segment and the initial depth map.

[0087] The second target depth value determination unit is used to determine the target depth value of adjacent boundary pixels based on the adjacent boundary pixels between adjacent bone segments and a preset depth gradient threshold.

[0088] A high-precision relative depth field construction subunit is used to construct a high-precision relative depth field based on the target depth values ​​of each pixel of the bone segment and the target depth values ​​of the adjacent boundary pixels.

[0089] Optional edge illumination reflection field building blocks include:

[0090] An edge pixel set determination unit is used to determine the edge pixel set of the portrait segmentation mask based on the boundary of the portrait segmentation mask and a preset edge band width;

[0091] An ambient light component determination unit is used to determine the ambient light component based on the brightness of neighboring pixels of the edge pixels in the edge pixel set of the portrait segmentation mask.

[0092] An edge normal vector determination unit is used to determine the edge normal vector corresponding to the edge pixel based on the edge gradient of the portrait segmentation mask.

[0093] The directional light component determination unit is used to determine the directional light component based on the edge normal vector and the direction of the main light source in the ambient lighting information;

[0094] A reflected light component determination unit is used to determine the reflected light component based on the color channel data of the edge pixels;

[0095] An edge illumination reflection field construction subunit is used to construct the edge illumination reflection field based on the ambient light component, directional light component, and reflected light component corresponding to each edge pixel.

[0096] Optionally, the pixel-by-pixel occlusion decision determination unit includes:

[0097] The query unit is used to query the skeletal semantic segment marker of the target pixel from the local skeletal semantic field;

[0098] The target bone segment determination unit is used to determine the target bone segment to which the target pixel belongs based on the bone semantic segment marker of the target pixel;

[0099] The depth value acquisition unit is used to acquire the target depth value of all pixels within the target bone segment from the high-precision relative depth field, and to acquire the scene depth value of the virtual scene at the target pixel position.

[0100] The segment-level aggregation depth determination unit is used to determine the segment-level aggregation depth based on the target depth values ​​of all pixels within the target bone segment;

[0101] An occlusion decision determination unit is used to determine the occlusion decision of the target pixel based on the segment-level aggregation depth and the scene depth value.

[0102] Optionally, the virtual scene information includes: virtual element position information, and the collision avoidance vector determination unit includes:

[0103] A virtual element vertex determination unit is used to determine the virtual element vertex based on the virtual element position information, and to determine the projection position of the virtual element vertex in the screen space.

[0104] A predicted displacement determination unit is used to determine the predicted displacement corresponding to the projected position based on the dynamic temporal motion field.

[0105] The collision avoidance vector determination subunit is used to determine the collision avoidance vector based on the predicted displacement, preset safety threshold and offset coefficient corresponding to the projected position.

[0106] Optionally, the virtual scene information includes: lighting parameters of the virtual engine, and the unit for transmitting lighting mapping determination includes:

[0107] The extraction unit is used to extract the ambient light component, directional light component, and reflected light component of the edge pixels from the edge illumination reflection field;

[0108] The original lighting parameter query unit is used to query the original lighting parameters corresponding to the spatial position of the edge pixel in the lighting parameters of the virtual engine.

[0109] The spatial attenuation weight determination unit is used to determine the spatial attenuation weight based on the edge pixels and the portrait segmentation mask;

[0110] The transfer illumination mapping determination subunit is used to determine the transfer illumination mapping based on the ambient light component, the directional light component, the reflected light component, the original illumination parameters, and the spatial attenuation weight.

[0111] Optionally, the target alpha mask determination unit includes:

[0112] The target neighbor pixel determination unit is used to determine the target neighbor pixel from the neighbor pixels of the target pixel based on the local skeleton semantic field and the dynamic temporal motion field.

[0113] A spatial smoothing constraint determination unit is used to determine spatial smoothing constraints based on the target pixel and the target neighboring pixels;

[0114] A depth continuity constraint determination unit is used to determine depth continuity constraints based on the high-precision relative depth field, the target pixel, and the neighboring pixels of the target pixel.

[0115] A pixel position determination unit is used to determine the corresponding pixel position of the target pixel in the previous frame based on the dynamic temporal motion field and the target pixel position of the current frame.

[0116] The temporal consistency constraint determination unit is used to determine the temporal consistency constraint based on the current frame alpha value of the target pixel and the previous frame alpha value of the corresponding pixel position.

[0117] The target alpha mask determination subunit is used to determine the target alpha mask based on the spatial smoothness constraint, the depth continuity constraint, and the temporal consistency constraint.

[0118] A third aspect of the present invention provides an electronic device, comprising:

[0119] One or more processors;

[0120] A storage device on which one or more programs are stored;

[0121] When the one or more programs are executed by the one or more processors, the one or more processors implement the hybrid rendering method as described in any one of the first aspects.

[0122] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hybrid rendering method as described in any one of the first aspects.

[0123] As can be seen from the above scheme, the present invention provides a hybrid rendering method and apparatus. It constructs a multi-dimensional semantic depth fusion field based on a live-action video stream, including a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge lighting reflection field. This multi-dimensional semantic depth fusion field provides corresponding information for occlusion processing, collision avoidance, edge lighting processing, and alpha mask determination. Pixel-by-pixel occlusion decisions are determined based on the local skeletal semantic field and the high-precision relative depth field, allowing occlusion relationships to combine skeletal semantics and depth information simultaneously, reducing erroneous occlusion and occlusion state jumps in cases of rapid movement and limb intersection. Collision avoidance vectors are determined based on the dynamic temporal motion field, enabling virtual elements to adjust their positions according to the real person's temporal motion information. Transmission lighting mapping is determined based on the edge lighting reflection field, ensuring a continuous transition between the real person's edge lighting and the virtual scene lighting. Simultaneously, target alpha masks are determined based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field, reducing flickering and ghosting at the virtual-real edges. This effectively improves the stability and continuity of hybrid rendering when real and virtual elements interact closely. Attached Figure Description

[0124] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0125] Figure 1 A flowchart of a hybrid rendering method provided in an embodiment of the present invention;

[0126] Figure 2 This is a schematic diagram of a hybrid rendering apparatus provided for another embodiment of the present invention. Detailed Implementation

[0127] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0128] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0129] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0130] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0131] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0132] This invention provides a hybrid rendering method, such as... Figure 1 As shown, the specific steps include:

[0133] S101. Obtain live video stream and virtual scene information.

[0134] The virtual scene information includes at least: virtual element position information and virtual engine lighting parameters. Virtual elements can include virtual characters, virtual props, virtual stage elements, and other renderable objects in the virtual scene, without limitation here. Virtual element position information is used to characterize the position of virtual elements in the virtual scene coordinate system, camera coordinate system, or screen space. Virtual engine lighting parameters refer to the configuration data of light sources in the virtual scene, including but not limited to the direction of the main light source, ambient light intensity, color temperature, shadow parameters, etc., without limitation here; live-action video stream refers to a sequence of video frames containing a live subject. The live subject can be performers, hosts, live streamers, or other personnel who need to be mixed with the virtual scene for rendering, without limitation here.

[0135] It should be noted that live video streams can originate from live on-site capture or from real-time network streaming. In live performance scenarios, one or more camera devices can be used to capture video data of the live subject, and hardware synchronization phase-locked loops can be used to ensure alignment of multiple frames. In remote live streaming scenarios, video streams sent by remote capture devices can be received, and frame synchronization can be ensured through buffer jitter compensation.

[0136] S102. Construct a multi-dimensional semantic deep fusion field based on real-person video streams.

[0137] Among them, the multidimensional semantic deep fusion field includes a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge illumination reflection field.

[0138] The local skeletal semantic field describes the human skeletal segments corresponding to different spatial locations. The high-precision relative depth field describes the depth information corresponding to each position of the real subject. The dynamic temporal motion field describes the predicted displacement information determined by the motion state of the real subject in consecutive frames. The edge illumination and reflection field describes the ambient light component, directional light component, and reflected light component corresponding to the edge position of the real subject.

[0139] Optionally, the above fields can be generated synchronously according to the video frames of the live video stream, so that the local skeletal semantic field, high-precision relative depth field, dynamic temporal motion field and edge illumination reflection field corresponding to the same video frame have corresponding time information.

[0140] Optionally, in another embodiment of the present invention, one implementation of step S102 includes the following steps (steps A1 to A7):

[0141] Step A1: Obtain the live video stream.

[0142] Step A2: Perform human pose estimation based on the live video stream to obtain the human pose estimation result, and perform human image segmentation on the live subject in the live video stream to obtain the human image segmentation mask.

[0143] Human pose estimation refers to the process of detecting and locating human skeletal joints from monocular or binocular video frames using deep learning networks.

[0144] Optionally, in the actual application of the present invention, a pre-trained pose estimation network (such as OpenPose, MediaPipe, etc.) can be used to process the live video stream frame by frame and output the two-dimensional or three-dimensional coordinates of K skeletal joints. The value of K can be configured as 21, 25 or 33 joints according to the accuracy requirements, and is not limited here.

[0145] Human face segmentation can, but is not limited to, using a real-time segmentation network (such as MODNet) to process the live video stream frame by frame and output a human face segmentation mask with the same resolution as the video frame. The value of each pixel in the mask is between 0 and 1, representing the confidence that the pixel belongs to the foreground region of the live person.

[0146] It should be noted that the portrait segmentation mask (Alpha Mask) plays a dual role in this scheme: firstly, it serves as the basis for extracting edge pixel positions when constructing the edge illumination reflection field; secondly, it serves as the initial input for subsequent spatiotemporal continuous Alpha mask optimization. It should also be noted that this invention does not limit the specific implementation of the upstream segmentation network, only requiring that its output mask format be aligned with the spatial coordinate system of the multidimensional semantic deep fusion field.

[0147] Step A3: Based on the human pose estimation results, construct a local skeletal semantic field.

[0148] Optionally, in another embodiment of the present invention, one implementation of step A3 specifically includes the following steps (steps A31 to A35):

[0149] Step A31: Determine the skeletal axis based on the spatial position of the skeletal joints.

[0150] Among them, the skeletal axis is a spatial line segment or spatial axis determined by two skeletal joints that have a joint connection relationship.

[0151] For example, the skeletal axis of the corresponding bone segment can be determined based on the spatial position of the parent and child joints.

[0152] Specifically, but not limited to, the line segment formed by connecting the coordinates of the parent and child joints can be used as the skeletal axis of the bone segment; this is not limited here.

[0153] Step A32: Determine the projection position of the spatial location to be processed on each bone axis.

[0154] Here, the spatial position to be processed refers to the position coordinates of the pixel or 3D point currently being calculated. The projected position refers to the perpendicular position coordinates of the spatial position to be processed on each bone axis.

[0155] Step A33: Based on the projection position and the spatial position to be processed, determine the semantic attribution strength of the spatial position to be processed for each bone segment.

[0156] In the practical application of this invention, for the spatial location to be processed It can be determined that it is in the first The projection position on each skeletal axis is used to determine the semantic attribution strength based on the distance between the spatial location to be processed and the projection position.

[0157] Optional, the first Semantic attribution strength of each bone segment It can be represented as:

[0158] ;in, For the first The skeletal axis of each bone segment for arrive The projection position, This is the preset spatial attenuation coefficient.

[0159] It is understandable that the smaller the distance between the spatial location to be processed and a certain skeletal axis, the stronger its semantic attribution strength for that skeletal segment.

[0160] Step A34: Based on the semantic attribution strength of each bone segment to the spatial location to be processed, determine the bone semantic segment label corresponding to the spatial location to be processed.

[0161] Continuing with the above example, after determining the semantic attribution strength of the spatial location to be processed for each skeletal segment, the maximum semantic attribution strength can be determined from among the semantic attribution strengths, and the skeletal segment corresponding to the maximum semantic attribution strength can be determined as the skeletal semantic segment corresponding to the spatial location to be processed. Accordingly, this can be expressed as:

[0162] ;

[0163] in, To determine the location of the space to be processed The corresponding skeletal semantic segment markers. This indicates the total number of bone segments.

[0164] Step A35: Construct a local skeletal semantic field based on the skeletal semantic segment tags corresponding to each spatial location to be processed.

[0165] Specifically, the local skeletal semantic field S is formed by the skeletal semantic segment markers corresponding to each spatial location. That is, S includes the skeletal semantic field S of each location within the spatial region corresponding to the real subject. .

[0166] This invention explicitly assigns skeletal semantic segment labels to each spatial location, enabling subsequent occlusion grouping decisions to obtain spatial semantic anchors. For example, when an arm moves from above a virtual prop to behind it, S maintains semantic consistency across every pixel in the arm region, allowing the subsequent occlusion decision system to recognize "this is the same arm" rather than "a series of unrelated depth points".

[0167] Step A4: Construct a high-precision relative depth field based on the initial depth map and local skeletal semantic field corresponding to the live video stream.

[0168] In a live performance scenario, the initial depth map can be raw depth data directly obtained from a depth sensor. In a remote live streaming scenario, the initial depth map can be pseudo-depth data output from a monocular depth estimation network, etc., without limitation. Both scenarios' initial depth maps suffer from noise and edge blurring. Therefore, in another embodiment of the invention, the invention utilizes the skeletal semantic information already established in S to aggregate and smooth the depth values ​​within the same bone segment. One implementation of step A4 specifically includes the following steps (steps A41 to A44):

[0169] Step A41: Based on the skeletal semantic segment labels in the local skeletal semantic field, divide the pixels in the initial depth map into pixel sets corresponding to different skeletal segments.

[0170] Specifically, for each pixel in the initial depth map, the corresponding skeletal semantic segment label in the local skeletal semantic field is queried, and all pixels with the same skeletal semantic segment label are grouped into the same pixel set. Thus, each skeletal segment corresponds to a pixel set, and the pixels within each set belong to the spatial range of that skeletal segment. The pixel set corresponding to each bone segment It can be represented as:

[0171] .

[0172] Step A42: Based on the pixel set corresponding to the bone segment and the initial depth map, determine the target depth value of each pixel in the bone segment.

[0173] For pixels belonging to the same bone segment, the initial depth values ​​within that pixel set can be aggregated to determine the target depth value for the corresponding bone segment. One possible implementation can be expressed as:

[0174] ;

[0175] in, is the initial depth value of pixel q in the initial depth map.

[0176] This invention uses skeletal semantic segments as smooth windows instead of fixed-size spatial windows, which eliminates depth noise within segments while automatically avoiding depth contamination across limbs.

[0177] Step A43: Determine the target depth value of the adjacent boundary pixels based on the adjacent boundary pixels between adjacent bone segments and the preset depth gradient threshold.

[0178] The adjacent boundary pixels between adjacent bone segments can be determined based on the local skeletal semantic field. For each adjacent boundary pixel, its depth value is adjusted to ensure that the depth gradient corresponding to the adjacent boundary pixel position is not less than a preset depth gradient threshold. The target depth value of the adjacent boundary pixels is obtained. One possible constraint relationship is:

[0179] ;

[0180] in, For skeletal segments and The set of adjacent boundary pixels.

[0181] This invention ensures sufficient depth discontinuity at bone segment boundaries through the aforementioned constraints, preventing the aggregation smoothing operation from blurring depth transitions at limb junctions. The final output high-precision relative depth field D maintains depth continuity within the limb, preserves clear depth transitions at limb boundaries, and eliminates noise in the initial depth map.

[0182] Step A44: Construct a high-precision relative depth field based on the target depth values ​​of each pixel in the skeletal segment and the target depth values ​​of adjacent boundary pixels.

[0183] Specifically, a high-precision relative depth field is constructed by combining the target depth values ​​of each pixel in the skeletal segment with the target depth values ​​of adjacent boundary pixels.

[0184] Step A5: Construct a dynamic temporal motion field based on the human pose estimation results of consecutive frames.

[0185] In one embodiment of the present invention, the motion trajectory of each skeletal joint in multiple consecutive video frames can be determined based on the human pose estimation results of consecutive frames. The motion trajectory is then input into a temporal prediction model (e.g., a lightweight Transformer decoder) to determine the predicted displacement position of each skeletal joint within the next time window Δt. After obtaining the predicted displacement position, it can be used to determine the displacement position. The displacement relative to the current frame position determines the dynamic temporal motion field. For the spatial position to be processed... The corresponding predicted displacement can be expressed as:

[0186] ;

[0187] in, for exist The position of the frame.

[0188] The values ​​corresponding to each position in the dynamic temporal motion field M are used to characterize the displacement information of the real subject from the current position to the predicted displacement position, so that subsequent processes can query the predicted motion information of the real subject based on the position of the virtual element. The physical meaning of M is "the spatial region that the real subject may reach in the next frame", which is used to drive the prediction and avoidance of the virtual element.

[0189] Step A6: Based on the human face segmentation mask and the live video stream, construct the edge illumination reflection field.

[0190] Optionally, in another embodiment of the present invention, one implementation of step A6 specifically includes the following steps (steps A61 to A64):

[0191] Step A61: Based on the boundary of the portrait segmentation mask and the preset edge band width, determine the set of edge pixels of the portrait segmentation mask.

[0192] In the practical application of the present invention, the boundary of the portrait segmentation mask can be determined first, and the preset edge band width can be extended to the inner and outer sides of the portrait segmentation mask based on the boundary to form an edge region. The pixels located in the edge region can form an edge pixel set.

[0193] Step A62: Determine the ambient light component based on the brightness of neighboring pixels of the edge pixels in the edge pixel set of the portrait segmentation mask.

[0194] Among them, the ambient light component can be composed of edge pixels. The brightness of neighboring pixels is determined by means such as average value or weighted average value (e.g., the weight decreases as the distance increases), but no restrictions are imposed here.

[0195] Step A63: Based on the edge gradient of the portrait segmentation mask, determine the edge normal vector corresponding to the edge pixel.

[0196] In the practical application of this invention, the edge normal vector corresponding to the edge pixel can be calculated from the edge gradient of the portrait segmentation mask, and this is not limited here.

[0197] Step A64: Determine the directional light component based on the direction of the main light source in the edge normal vector and ambient lighting information.

[0198] The direction of the main light source in the ambient lighting information can be output by the ambient lighting probe, and there is no limitation here.

[0199] In the practical application of this invention, the directional light component Based on the direction of the main light source With edge normal vector The dot product is calculated to obtain, i.e. .

[0200] Step A65: Determine the reflected light components based on the color channel data of the edge pixels.

[0201] In practical applications of this invention, the reflected light component can be obtained by performing specular detection on the color channel data of edge pixels. For example, local specular peaks in the RGB channels can be detected, and the reflected light component can be determined based on the specular peaks.

[0202] Step A66: Construct the edge illumination reflection field based on the ambient light component, directional light component, and reflected light component corresponding to each edge pixel.

[0203] Specifically, combining edge pixels Corresponding ambient light component directional light component and reflected light components Generate edge pixels Edge lighting information , Based on all edge pixels Edge lighting information Generate an edge illumination reflection field L.

[0204] Step A7: Construct a multi-dimensional semantic depth fusion field based on the local skeletal semantic field, high-precision relative depth field, dynamic temporal motion field, and edge illumination reflection field.

[0205] This invention constructs a frame-by-frame synchronous field, generating four field images {S,D,M,L} independently for each frame, which are then combined to form a multi-dimensional semantic deep fusion field F={S,D,M,L}. It should be noted that the construction process of F in this invention does not involve virtual scene rendering at all, and is therefore transparent to the upstream input source (multiple on-site cameras / remote green screen).

[0206] S103. Based on the local skeletal semantic field and the high-precision relative depth field, determine the pixel-by-pixel occlusion decision.

[0207] Among them, pixel-by-pixel occlusion decision is used to characterize the front-to-back occlusion relationship between the real subject and the virtual scene at the target pixel location.

[0208] First, it should be noted that in traditional solutions, occlusion decisions are made based on pixel-by-pixel depth comparison: the depth value of the real person is compared with the Z-buffer of the virtual scene, with the closer value occluding the farther one. In a typical virtual concert scenario, when an artist moves their arm from in front of a virtual prop to behind it in a semi-sideways posture, the local depth of the arm may be confused with the depth of the torso, causing some pixels of the same arm to penetrate the prop while others are incorrectly occluded. Therefore, this invention employs a local skeletal semantic field and a high-precision relative depth field to determine pixel-by-pixel occlusion decisions. In another embodiment of this invention, one implementation of step S103 specifically includes the following steps (steps B1 to B5):

[0209] Step B1: Query the skeletal semantic segment markers (such as left upper arm, right forearm, etc.) of the target pixel from the local skeletal semantic field.

[0210] The target pixel refers to the screen pixel for which occlusion relationship needs to be determined.

[0211] Step B2: Determine the target bone segment to which the target pixel belongs based on the skeletal semantic segment label of the target pixel.

[0212] Step B3: Obtain the target depth values ​​of all pixels within the target bone segment from the high-precision relative depth field, and obtain the scene depth values ​​of the virtual scene at the target pixel position.

[0213] Step B4: Determine the segment-level aggregation depth based on the target depth values ​​of all pixels within the target bone segment.

[0214] In the practical application of this invention, the segment-level aggregation depth It can be calculated using the following formula:

[0215] .

[0216] This invention aggregates depth information according to bone segments, enabling pixels belonging to the same bone segment to use corresponding segment-level depth information for occlusion determination, thereby reducing changes in occlusion state caused by changes in a single depth pixel.

[0217] Step B5: Determine the occlusion decision for the target pixel based on the segment-level aggregation depth and scene depth value.

[0218] In the practical application of this invention, the occlusion decision of the target pixel can be determined using the following formula. :

[0219] ;

[0220] in, This represents the scene depth value. This is the segment-level aggregation depth calculated based on the bone segment to which pixel p belongs.

[0221] This invention uses segment-level aggregated depth instead of pixel-level original depth for occlusion comparison. All pixels within the same bone segment share consistent occlusion decisions, avoiding the flipping of occlusion states within the same segment due to depth noise during pixel-by-pixel depth comparison (flickering of half an arm being penetrated and half being occluded).

[0222] S104. Determine the collision avoidance vector based on the dynamic temporal motion field.

[0223] Among them, the collision avoidance vector is a displacement vector used to adjust the position or motion state of a virtual element.

[0224] Existing solutions rely on manual choreography or post-processing bounding box detection to handle collisions between virtual and real elements, frequently resulting in clipping and movement conflicts in fast-paced dance scenes. Traditional bounding box solutions pre-define colliders for various parts of the artist's body and detect the geometric intersections between these colliders and virtual elements (virtual dancers, props). The problem is that the detection itself is lagging; by the time the system detects an intersection between an arm and a virtual dancer, clipping has already occurred.

[0225] Optionally, in another embodiment of the present invention, one implementation of step S104 specifically includes the following steps (steps C1 to C3):

[0226] Step C1: Based on the virtual element position information, determine the virtual element vertices and the projection position of the virtual element vertices in screen space.

[0227] Specifically, based on the model information of the virtual element, the multiple virtual element vertices constituting the virtual element are obtained, and the virtual element vertices are mapped to the screen space according to the projection relationship corresponding to the virtual camera to obtain the corresponding projection positions. .

[0228] Step C2: Determine the predicted displacement corresponding to the projection position based on the dynamic temporal motion field.

[0229] For a virtual element vertex v, its projected position can be queried in a dynamic temporal motion field. Corresponding predicted displacement .

[0230] Step C3: Determine the collision avoidance vector based on the predicted displacement corresponding to the projection position, the preset safety threshold, and the offset coefficient.

[0231] First, the degree of spatial conflict between the virtual element vertices and the predicted motion area of ​​the real person is determined based on the predicted displacement corresponding to the projection position. :

[0232] ;

[0233] Then, when the degree of spatial conflict Greater than the preset safety threshold At that time, the collision avoidance vector can be determined based on the expected displacement and offset coefficient. When the degree of spatial conflict Not greater than the preset safety threshold At that time, the collision avoidance vector can be... It is determined to be the zero vector. One possible way to represent it is as follows:

[0234] ;in, Offset coefficient (0 < <1), used to control the avoidance range.

[0235] This invention applies a bias to virtual vertices located on the real person's movement path, with the bias direction consistent with the real person's movement direction, making the avoidance action natural and smooth. The virtual element motion control system allows virtual dancers or stage elements to smoothly shift in a safe direction before an actual collision occurs.

[0236] S105. Determine the transmitted illumination map based on the edge illumination reflection field.

[0237] Optionally, in another embodiment of the present invention, one implementation of step S105 specifically includes the following steps (steps D1 to D4):

[0238] Step D1: Extract the ambient light component, directional light component, and reflected light component of the edge pixels from the edge illumination reflection field.

[0239] Step D2: Query the original lighting parameters corresponding to the spatial position of the edge pixels in the lighting parameters of the virtual engine.

[0240] Step D3: Determine the spatial attenuation weights based on edge pixels and portrait segmentation mask.

[0241] In the practical application of this invention, spatial attenuation weight It can be calculated based on the following formula:

[0242] ;

[0243] in, , The preset fusion transition bandwidth parameters for the system. The normal distance (in pixels) from edge pixel q to the edge boundary of the portrait segmentation mask.

[0244] It should be noted that, This is used to control the normal distance bandwidth required for the transition from using entirely realistic lighting to using entirely virtual engine lighting. The larger the size, the smoother the transition. The smaller the size, the steeper the transition. The value is on the same order of magnitude as the preset edge band width, typically ranging from several to tens of pixels, ensuring that the blending transition occurs within the neighborhood of the virtual and real boundaries without spilling out to other areas of the image.

[0245] Step D4: Determine the transmitted illumination map based on the ambient light component, directional light component, reflected light component, original illumination parameters, and spatial attenuation weight.

[0246] In the practical application of this invention, the following formula can be used to determine the transmitted illumination mapping. :

[0247] ;

[0248] in, To extract edge illumination information of edge pixels from the edge illumination reflection field, including ambient light component, directional light component and reflected light component; These are the original lighting parameters.

[0249] Traditional virtual-real compositing schemes treat portrait separation and rendering as two completely decoupled processes. Portrait separation refers to any process of extracting foreground pixels from a video portrait. In real-time green screen scenes, this manifests as chroma keying, while in real-time live scenes, it manifests as depth map-based foreground segmentation. Although the algorithms differ, they are functionally equivalent, resulting in a binary (or multi-valued alpha) mask of the portrait foreground and a transparent background. Traditional methods discard pixel-level ambient lighting information during the separation stage, retaining only RGB colors; the rendering stage uses the virtual engine's global illumination parameters, with no data interaction between the two. This decoupling leads to a data loss: the ambient light actually received by the portrait edges is inconsistent with the virtual engine's preset global illumination.

[0250] This invention achieves progressive fusion of lighting at the boundary between virtual and real light, with the inner edge entirely utilizing edge lighting information extracted from the edge lighting reflection field. It decays exponentially outwards along the normal. Gradually switch to the virtual engine's original lighting parameters The transition from hard switching to smooth transition eliminates abrupt changes in light and shadow at the junction of virtual and real elements.

[0251] S106. Determine the target alpha mask based on the local skeletal semantic field, high-precision relative depth field, and dynamic temporal motion field.

[0252] It should be noted that, Figure 1 The current illustration is only based on the order of steps S103 to S106. In the actual application of this invention, the order is not limited and they can be executed simultaneously.

[0253] Optionally, in another embodiment of the present invention, one implementation of step S106 specifically includes the following steps (steps E1 to E6):

[0254] Step E1: Based on the local skeletal semantic field and dynamic temporal motion field, determine the target neighboring pixels from the neighboring pixels of the target pixel.

[0255] Among them, the target neighboring pixel is the neighboring pixel that belongs to the same bone segment as the target pixel and is located in the direction of the target pixel's movement.

[0256] For target pixel Target neighboring pixel set It can be represented as:

[0257] ;

[0258] in, The unit vector of motion direction extracted from the dynamic temporal motion field M. The neighboring pixels of the target pixel. and These represent the skeletal semantic segment tags corresponding to the target pixel and its adjacent pixels, respectively.

[0259] Step E2: Determine spatial smoothing constraints based on the target pixel and its neighboring pixels.

[0260] In the practical application of this invention, the spatial smoothing constraint can be determined using the following formula. :

[0261] ;in, The alpha value of the target pixel. The alpha value is the alpha value of the neighboring pixels of the target.

[0262] This invention prevents alpha discontinuity within a segment by constraining adjacent pixels along the motion direction to have consistent alpha values.

[0263] Step E3: Determine depth continuity constraints based on the high-precision relative depth field, the target pixel, and the neighboring pixels of the target pixel.

[0264] In the practical application of this invention, the following formula can be used to determine the depth continuity constraint. :

[0265] ;

[0266] in, N(p) is the set of neighboring pixels of the target pixel p (e.g., the pixels in the 8-neighborhood of pixel p). The system has a preset depth difference sensitivity parameter, whose unit is consistent with the depth value of the high-precision relative depth field, which controls the attenuation bandwidth of depth difference on the Alpha smoothing constraint. The larger the value, the more forced the alpha similarity is to adjacent pixels with large depth differences, resulting in stronger constraints. The smaller the value, the more only adjacent pixels with minimal depth difference are forced to accept the constraint, resulting in a weaker constraint. The typical value of is on the same order of magnitude as the noise standard deviation of the depth value of the high-precision relative depth field, thus ensuring that the alpha of surfaces at the same depth is consistent while the alpha of planes at different depths separates at the boundary.

[0267] Step E4: Based on the dynamic temporal motion field and the target pixel position in the current frame, determine the corresponding pixel position of the target pixel in the previous frame.

[0268] This invention establishes the positional relationship between the target pixel in the current frame and the corresponding pixel in the previous frame using a dynamic temporal motion field. For the target pixel position p in the current frame, the position of the corresponding pixel in the previous frame can be determined based on the motion displacement in the dynamic temporal motion field, and the Alpha value of that corresponding pixel position in the previous frame can be read.

[0269] Step E5: Determine the temporal consistency constraints based on the current frame alpha value of the target pixel and the previous frame alpha value of the corresponding pixel position.

[0270] In the practical application of this invention, timing consistency constraints can be determined based on the following formula. :

[0271] ;

[0272] in, The target alpha mask optimized in step E6 for the previous frame (time t-1), i.e., the final alpha value matrix at time t-1, is compared with... The corresponding time index is t-1, which is used as the pixel index function in the temporal consistency constraint formula. The formula in step E6 is used as the overall mask, meaning that the output is optimized using the formula in step E6 for each frame. As a timing consistency constraint for the next frame The input reference forms a frame-by-frame timing closed loop. It is a scalar; summing over all pixels p compresses the N×M dimensional α-mask energy into a 1×1 scalar. The formula contains a scalar difference at the single-pixel level, not a matrix operation, and the square is the ordinary square of that scalar difference.

[0273] This invention aligns the pixel correspondence between consecutive frames using motion vectors and then compares the alpha differences to eliminate edge flickering caused by independent frame-by-frame segmentation.

[0274] Step E6: Determine the target alpha mask based on spatial smoothness constraints, depth continuity constraints, and temporal consistency constraints.

[0275] In the practical application of this invention, the optimization objective is to minimize the weighted energy of three constraints (spatial smoothness constraint, depth continuity constraint, and temporal consistency constraint) to determine the target alpha mask.

[0276] In the practical application of this invention, the target alpha mask can be determined using the following formula. :

[0277] ;

[0278] in, , , These are the weighting coefficients, and The adjustment is dynamically determined by the scene type (on-site / remote) and the interaction distance. For example, it increases during close-range interaction. , To increase the weights of spatial smoothness constraints and temporal consistency constraints; increase the weights for long distances. It relies on depth continuity constraints.

[0279] This invention solves spatial smoothness constraints, depth continuity constraints, and temporal consistency constraints together to output an Alpha mask that is doubly smooth in both space and time, effectively eliminating flicker and ghosting.

[0280] S107. Based on pixel-wise occlusion decision-making, collision avoidance vectors, transitive lighting mapping, and target alpha masking, perform virtual-real hybrid rendering of live video streams and virtual scenes.

[0281] In this embodiment, pixel-by-pixel occlusion decision can be provided to the pixel blending stage of the virtual engine to determine the front-to-back relationship between real-person video pixels and virtual scene pixels at different pixel positions; collision avoidance vectors can be provided to the motion control process of virtual elements to adjust the position of virtual elements; transfer lighting mapping can be provided to the pixel shading process to adjust the lighting parameters corresponding to the virtual-real edge regions; and target alpha masking can be used to perform alpha blending of real-person video pixels and virtual scene pixels.

[0282] Therefore, occlusion processing, virtual element motion processing, edge lighting processing, and alpha blending can all use data provided by the multidimensional semantic depth fusion field, so that each processing result corresponds to the semantic, depth, motion, and edge lighting state of the same real-person video frame.

[0283] As can be seen from the above scheme, this invention provides a hybrid rendering method that constructs a multi-dimensional semantic depth fusion field based on a live-action video stream, including a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge lighting reflection field. This multi-dimensional semantic depth fusion field provides corresponding information for occlusion processing, collision avoidance, edge lighting processing, and alpha mask determination. Pixel-by-pixel occlusion decisions are determined based on the local skeletal semantic field and the high-precision relative depth field, allowing occlusion relationships to combine skeletal semantics and depth information simultaneously, reducing erroneous occlusion and occlusion state jumps in cases of rapid movement and limb overlap. Collision avoidance vectors are determined based on the dynamic temporal motion field, enabling virtual elements to adjust their positions according to the real person's temporal motion information. Transmission lighting mapping is determined based on the edge lighting reflection field, ensuring a continuous transition between the real person's edge lighting and the virtual scene lighting. Simultaneously, target alpha masks are determined based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field, reducing flickering and ghosting at the virtual-real edges. This effectively improves the stability and continuity of hybrid rendering when real and virtual elements interact closely.

[0284] Another embodiment of the present invention provides a hybrid rendering apparatus, such as Figure 2 As shown, it specifically includes:

[0285] Acquisition unit 201 is used to acquire live video streams and virtual scene information.

[0286] The virtual scene information includes at least: virtual element location information and virtual engine lighting parameters.

[0287] The multidimensional semantic deep fusion field construction unit 202 is used to construct a multidimensional semantic deep fusion field based on real-person video streams.

[0288] Among them, the multidimensional semantic deep fusion field includes a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge illumination reflection field.

[0289] The pixel-by-pixel occlusion decision determination unit 203 is used to determine pixel-by-pixel occlusion decisions based on the local skeletal semantic field and the high-precision relative depth field.

[0290] The collision avoidance vector determination unit 204 is used to determine the collision avoidance vector based on the dynamic temporal motion field and the virtual element position information.

[0291] The transit lighting mapping determination unit 205 is used to determine the transit lighting mapping based on the edge lighting reflection field and the lighting parameters of the virtual engine.

[0292] The target alpha mask determination unit 206 is used to determine the target alpha mask based on the local skeletal semantic field, the high-precision relative depth field and the dynamic temporal motion field.

[0293] The virtual-real hybrid rendering unit 207 is used to perform virtual-real hybrid rendering on live video streams and virtual scenes based on pixel-by-pixel occlusion decision, collision avoidance vector, transitive lighting mapping and target alpha mask.

[0294] For details on the specific operation of the units disclosed in the above embodiments of the present invention, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.

[0295] As can be seen from the above scheme, the present invention provides a hybrid rendering device that constructs a multi-dimensional semantic depth fusion field based on a live-action video stream, including a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge lighting reflection field. This multi-dimensional semantic depth fusion field provides corresponding information for occlusion processing, collision avoidance, edge lighting processing, and alpha mask determination. Pixel-by-pixel occlusion decisions are determined based on the local skeletal semantic field and the high-precision relative depth field, allowing occlusion relationships to combine skeletal semantics and depth information simultaneously, reducing erroneous occlusion and occlusion state jumps in cases of rapid movement and limb intersection. Collision avoidance vectors are determined based on the dynamic temporal motion field, enabling virtual elements to adjust their positions according to the real person's temporal motion information. Transmission lighting mapping is determined based on the edge lighting reflection field, ensuring a continuous transition between the real person's edge lighting and the virtual scene lighting. Simultaneously, target alpha masks are determined based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field, reducing flickering and ghosting at the virtual-real edges. This effectively improves the stability and continuity of hybrid rendering when real and virtual elements interact closely.

[0296] Another embodiment of the present invention provides an electronic device, comprising:

[0297] One or more processors.

[0298] A storage device on which one or more programs are stored.

[0299] When the one or more programs are executed by the one or more processors, the one or more processors implement the hybrid rendering method as described in the above embodiments.

[0300] Another embodiment of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hybrid rendering method as described in the above embodiments.

[0301] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0302] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0303] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0304] Another embodiment of the present invention provides a computer program product, which, when executed, is used to perform the above-described hybrid rendering method.

[0305] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of the present invention.

[0306] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in this invention is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms for implementing the invention.

[0307] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0308] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions in the present invention.

Claims

1. A hybrid rendering method, characterized in that, include: Acquire live video streams and virtual scene information; Based on the live video stream, a multidimensional semantic deep fusion field is constructed; wherein, the multidimensional semantic deep fusion field includes a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge illumination reflection field; Based on the local skeletal semantic field and the high-precision relative depth field, a pixel-by-pixel occlusion decision is determined; Based on the dynamic temporal motion field, a collision avoidance vector is determined; Based on the edge illumination reflection field, determine the transmitted illumination map; Based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field, the target alpha mask is determined. Based on the pixel-by-pixel occlusion decision, the collision avoidance vector, the transitive lighting mapping, and the target alpha mask, the live video stream and the virtual scene are rendered using a hybrid rendering method.

2. The hybrid rendering method according to claim 1, characterized in that, The construction of a multi-dimensional semantic deep fusion field based on the live video stream includes: Obtain live video stream; Human pose estimation is performed based on the live video stream to obtain the human pose estimation result, and human figure segmentation is performed on the live subject in the live video stream to obtain the human figure segmentation mask. Based on the human pose estimation results, a local skeletal semantic field is constructed; Based on the initial depth map corresponding to the live video stream and the local skeletal semantic field, a high-precision relative depth field is constructed; Based on the human pose estimation results of consecutive frames, a dynamic temporal motion field is constructed; Based on the portrait segmentation mask and the live video stream, an edge illumination reflection field is constructed; A multidimensional semantic depth fusion field is constructed based on the local skeletal semantic field, the high-precision relative depth field, the dynamic temporal motion field, and the edge illumination reflection field.

3. The hybrid rendering method according to claim 2, characterized in that, The human pose estimation result includes the spatial location of skeletal joints. The construction of a local skeletal semantic field based on the human pose estimation result includes: Determine the skeletal axis based on the spatial location of skeletal joints; Determine the projection position of the spatial location to be processed on each bone axis; Based on the projection position and the spatial position to be processed, determine the semantic attribution strength of the spatial position to be processed for each bone segment; Based on the semantic attribution strength of each bone segment to the spatial location to be processed, the semantic segment label of the bone corresponding to the spatial location to be processed is determined. Based on the skeletal semantic segment tags corresponding to each of the spatial locations to be processed, a local skeletal semantic field is constructed.

4. The hybrid rendering method according to claim 2, characterized in that, The construction of a high-precision relative depth field based on the initial depth map corresponding to the live video stream and the local skeletal semantic field includes: Based on the bone semantic segment markers in the local bone semantic field, the pixels in the initial depth map are divided into pixel sets corresponding to different bone segments. Based on the pixel set corresponding to the bone segment and the initial depth map, the target depth value of each pixel of the bone segment is determined; The target depth value of the adjacent boundary pixels is determined based on the adjacent boundary pixels between adjacent bone segments and a preset depth gradient threshold. A high-precision relative depth field is constructed based on the target depth values ​​of each pixel in the bone segment and the target depth values ​​of the adjacent boundary pixels.

5. The hybrid rendering method according to claim 2, characterized in that, The construction of the edge illumination reflection field based on the portrait segmentation mask and the live video stream includes: Based on the boundary of the portrait segmentation mask and the preset edge band width, the set of edge pixels of the portrait segmentation mask is determined; The ambient light component is determined based on the brightness of the neighboring pixels of the edge pixels in the edge pixel set of the portrait segmentation mask. Based on the edge gradient of the portrait segmentation mask, the edge normal vector corresponding to the edge pixel is determined; Based on the edge normal vector and the direction of the main light source in the ambient lighting information, the directional light component is determined; The reflected light component is determined based on the color channel data of the edge pixels; The edge illumination reflection field is constructed based on the ambient light component, directional light component, and reflected light component corresponding to each edge pixel.

6. The hybrid rendering method according to claim 1, characterized in that, The step of determining pixel-by-pixel occlusion decisions based on the local skeletal semantic field and the high-precision relative depth field includes: Query the skeletal semantic segment marker of the target pixel from the local skeletal semantic field; Based on the skeletal semantic segment marker of the target pixel, the target bone segment to which the target pixel belongs is determined; The target depth values ​​of all pixels within the target bone segment are obtained from the high-precision relative depth field, and the scene depth value of the virtual scene at the target pixel position is obtained. Based on the target depth values ​​of all pixels within the target bone segment, determine the segment-level aggregation depth; Based on the segment-level aggregation depth and the scene depth value, the occlusion decision of the target pixel is determined.

7. The hybrid rendering method according to claim 1, characterized in that, The virtual scene information includes: virtual element position information; the determination of the collision avoidance vector based on the dynamic temporal motion field includes: Based on the virtual element position information, determine the virtual element vertices and the projection position of the virtual element vertices in screen space. The predicted displacement corresponding to the projection position is determined based on the dynamic temporal motion field. Based on the predicted displacement, preset safety threshold, and offset coefficient corresponding to the projected position, a collision avoidance vector is determined.

8. The hybrid rendering method according to claim 5, characterized in that, The virtual scene information includes: lighting parameters of the virtual engine; the determination of the transfer lighting map based on the edge lighting reflection field includes: Extract the ambient light component, directional light component, and reflected light component of the edge pixels from the edge illumination reflection field; Query the original lighting parameters corresponding to the spatial position of the edge pixel in the lighting parameters of the virtual engine; Based on the edge pixels and the portrait segmentation mask, determine the spatial attenuation weight; Based on the ambient light component, the directional light component, the reflected light component, the original illumination parameters, and the spatial attenuation weight, the transmitted illumination map is determined.

9. The hybrid rendering method according to claim 1, characterized in that, The determination of the target alpha mask based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field includes: Based on the local skeletal semantic field and the dynamic temporal motion field, the target neighboring pixels are determined from the neighboring pixels of the target pixel; Based on the target pixel and the target neighboring pixels, determine the spatial smoothness constraint; Based on the high-precision relative depth field, the target pixel, and the neighboring pixels of the target pixel, depth continuity constraints are determined; Based on the dynamic temporal motion field and the target pixel position in the current frame, determine the corresponding pixel position of the target pixel in the previous frame; Based on the current frame alpha value of the target pixel and the previous frame alpha value of the corresponding pixel position, a temporal consistency constraint is determined. The target alpha mask is determined based on the spatial smoothness constraint, the depth continuity constraint, and the temporal consistency constraint.

10. A hybrid rendering apparatus, characterized in that, include: The acquisition unit is used to acquire live video streams and virtual scene information; A multidimensional semantic deep fusion field construction unit is used to construct a multidimensional semantic deep fusion field based on the real-person video stream; wherein, the multidimensional semantic deep fusion field includes a local skeletal semantic field, a high-precision relative depth field, a dynamic temporal motion field, and an edge illumination reflection field; A pixel-by-pixel occlusion decision determination unit is used to determine a pixel-by-pixel occlusion decision based on the local skeletal semantic field and the high-precision relative depth field. The collision avoidance vector determination unit is used to determine the collision avoidance vector based on the dynamic temporal motion field. A transmission illumination mapping determination unit is used to determine the transmission illumination mapping based on the edge illumination reflection field; The target alpha mask determination unit is used to determine the target alpha mask based on the local skeletal semantic field, the high-precision relative depth field, and the dynamic temporal motion field. The virtual-real hybrid rendering unit is used to perform virtual-real hybrid rendering on the live video stream and the virtual scene based on the pixel-by-pixel occlusion decision, the collision avoidance vector, the transitive lighting map, and the target alpha mask.

Images