New viewpoint synthesis method for video conference scene

Abstract

The invention provides a new viewpoint synthesis method for a video conference system, and belongs to the field of computer vision. Under the condition of large parallax, sparse viewpoints are used as input, and high-quality dense new viewpoint reconstruction is carried out on a video conference scene through an image processing technology and a deep learning network. The method does not need extra training for unseen scenes, and has a complete generalization ability; through a mathematical model and an algorithm which are clear and have geometric significance, training acceleration is realized by utilizing a complete explicit expression mode, and meanwhile, the adaptability and the calculation efficiency of a video conference system are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to a new viewpoint synthesis method for a video conference scene. Background Art

[0002] With the continuous development of computer technology and communication technology, the requirements for the quality and applicability of video conferencing technology have increased significantly. The way of remote collaboration and communication based on communication technology can effectively save communication costs and improve people's work efficiency. Video conferencing technology has been widely used in different fields such as education, work, interpersonal communication and entertainment, and the market demand is growing. Among them, 3D video conferencing system, as one of the most promising applications of video conferencing technology, further improves the immersive experience of video conferencing system.

[0003] However, despite the rapid development of new viewpoint synthesis techniques in recent years, there are still limited methods to optimize 3D video conferencing systems. 3D video conferencing requires not only high-quality single-frame dense viewpoint rendering, but also effective control of the temporal consistency of the generated viewpoints (i.e., the coherence of the video) to further enhance the immersive experience and fidelity of the 3D video. Although the current mainstream methods have achieved significant progress by introducing new scene expressions, there is still considerable room for improvement in fidelity and temporal consistency.

[0004] Neural radiance fields are an implicit scene expression method based on scene volume rendering. Reference 1: Mildenhall B, Srinivasan PP, Tancik M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis [J]. Communications of the ACM, 2021, 65 (1): 99-106. The implicit scene expression was first introduced and high-quality new viewpoint reconstruction results were obtained. Compared with traditional geometric 3D reconstruction, the implicit scene expression method has significantly improved the scene fidelity, but it still has shortcomings in training and reasoning speed, denoising and deblurring process, and sparse view input scenes, which limits its widespread use in practical applications. Although the scene generalization research based on implicit scene expression has reduced the training time of cross-scene expression in recent years, it still takes a long time to achieve high-quality synthesis results. In addition, for scenes involving large parallax and sparse input, the existing implicit scene expression has unstable synthesis quality. These factors together limit the application of implicit scene expression in the field of 3D video conferencing.

[0005] The 3D Gaussian splatting method is an explicit 3D reconstruction method that uses anisotropic 3D Gaussian functions as scene representation. Reference 2: Kerbl B, Kopanas G, Leimkühler T, et al. 3D Gaussian Splatting for Real-Time Radiance Field Rendering [J]. ACM Trans. Graph., 2023, 42 (4): 139: 1-139: 14. Combining the advantages of gradient-based optimization and explicit structured data storage, high-quality real-time rendering can be achieved. However, this method relies on adaptive density-controlled optimization methods to fit the scene, which limits it to single-scene optimization. Therefore, many extension methods based on 3D Gaussian splatting have been proposed. These methods improve the expression ability under sparse viewpoint input and effectively solve the limitations of the original 3D Gaussian splatting, especially in the fields of scene generalization and dynamic scene expression.

[0006] The synthesis of new viewpoints in video conferencing scenarios often faces the following difficulties: 1) Large parallax problem. In video conferencing, the distance between the user and the camera is usually close. This leads to large parallax between the captured viewpoints (i.e., the difference in images seen from different perspectives), which in turn leads to complex occlusion relationships, difficulty in depth estimation, and increased image transformation errors. 2) Sparse input problem. For cost considerations, 3D video conferencing systems in commercial applications often limit the number of cameras. Therefore, the synthesis of new viewpoints often needs to rely on sparse input viewpoints. 3) Video conferencing scenarios require not only high-quality single-frame synthesis results, but also the overall quality of the new viewpoint video. This means that the synthesis method needs to maintain temporal consistency between different frames while ensuring the quality of a single frame. When faced with sparse input and large parallax, current methods often produce artifacts (i.e., false or unnatural parts in the image) in the rendered new viewpoint due to insufficient information. In addition, the temporal consistency of the synthesized video is often not ideal. In summary, the new viewpoint synthesis method in the video conferencing scenario needs to solve the following key problems at the same time:

[0007] 1) Ensure high quality synthesis under large parallax conditions

[0008] 2) Adapting to sparse input

[0009] 3) Maintaining the temporal consistency of the video

[0010] Given the limitations of existing technologies, this field still needs further optimization and innovation. Summary of the invention

[0011] In order to solve the problem of poor temporal consistency of new viewpoint video synthesis when sparse input viewpoints are used under large parallax, the present invention proposes a new viewpoint synthesis method for video conferencing scenes. Under large parallax conditions, sparse viewpoint images are used as input, and high-quality dense new viewpoint reconstruction of video conferencing scenes is performed through image processing technology and deep learning networks. The present invention does not require additional training for unseen scenes and has full generalization capabilities; through clear and geometrically meaningful mathematical models and algorithms, training acceleration is achieved using a fully explicit expression method, while improving the adaptability and computational efficiency of video conferencing scenes.

[0012] The steps of the specific technical solution of the present invention are as follows:

[0013] Step S1: Use a binocular depth estimation method to predict the input viewpoint image to obtain a depth map D' s ;

[0014] Step S2: Reduce the scene depth range to adjust the depth map D' s Perform fine correction to obtain the corrected depth map D s ;

[0015] Step S3: According to D s Calculate the target viewpoint depth map D t ;

[0016] Step S4: Calculate the projection image W of the input viewpoint image s and Plane Sweep Volume (PSV);

[0017] Step S5: constructing a fusion network;

[0018] Step S6, obtaining the fusion weight and opacity of PSV;

[0019] Step S7: Obtain the target viewpoint image I using the alpha fusion algorithm t .

[0020] In step S1, the depth map D' s The prediction process is as follows:

[0021] Step S11, a binocular camera is used to capture two input viewpoint images, the binocular camera includes an input camera and a target camera (do not write a closed-form expression), the two input viewpoint images are I1 and I2 respectively, image features of I1 and I2 are extracted through a feature extraction network, and the image features are F1 and F2 respectively; the feature extraction network uses a transformer network;

[0022] Step S12, through depth sequence transformation, obtain the image features of F2 under the corresponding camera posture of F1 The depth sequence transformation process is as follows:

[0023] First, the scene depth range is manually given, the given depth range is 200mm-5000mm, within the given depth range, the depth sequence is a i , i=1,2,…,N, the value range of N is 32≤N≤128; the grid coordinates are calculated according to the depth sequence, and the grid coordinate transformation process is:

[0024]

[0025] Where H(·) represents the transformation of homogeneous grid coordinates, is the grid coordinate of F1 corresponding to the camera pose, G 2d is the grid coordinate of F2 corresponding to the camera pose, K1 represents the intrinsic parameter matrix of I1 corresponding to the camera, K2 represents the intrinsic parameter matrix of I2 corresponding to the camera, E1 represents the extrinsic parameter matrix of I1 corresponding to the camera, E2 represents the extrinsic parameter matrix of I2 corresponding to the camera, H and W represent the height and width of F2;

[0026] Then, normalize Decompose H×W×2 into two H×W×1 shapes and Will Divide by W, and we get Will Dividing by H, we get Then re- and Splice into a grid coordinate with a shape of H×W×2 The normalized

[0027]

[0028] Finally, the normalized Input the grid_sample function of the PyTorch network together with F1 to output image features

[0029] Step S13, calculating the image feature correlation C i ; The image feature correlation C i The calculation formula is as follows:

[0030]

[0031] Where D is the dimension of the feature number channel of F1;

[0032] Step S14: according to the image feature correlation C i Calculate the depth map D' s; The depth map D' s The calculation process is as follows:

[0033] First, C i Set, get C d =[C1,C2,…,C N ];

[0034] Then, C d After normalization, we get M d , the normalized formula is: M d = SOFTMAX(C d );

[0035] Finally, PyTorch’s propagation mechanism is used to propagate M d With A d Multiply them to get the depth map D' of the input viewpoint image s , where s represents different input viewpoint images, A d for a i The depth sequence A d =[a1,a2,…,a N ].

[0036] In step S2, the fine correction process is as follows:

[0037] Step S21, narrow the depth range of step S12 to 200mm-500mm, and repeat steps S12 to S14; during the process, the depth sequence diagram b i The collection is B d =[b1,b2,…,b N ], the transformed image features are

[0038] Step S22, calculate N The variance Q with F1 d , Q d Input into the U-Net network to get the depth probability P d , according to formula D s =B d ·ρ d , get the corrected input viewpoint image depth map D s .

[0039] In step S3, the target viewpoint depth map D t The calculation method is as follows:

[0040] Since the depth map D s Contains the corresponding depth of each pixel in the input viewpoint image, D s (u s ,v s) After back-projection, we get the 3D point cloud O s (u s ,v s ), back projection calculation formula:

[0041]

[0042] in, is the inverse of the camera extrinsic augmented matrix of the input image; The inverse of the camera intrinsic parameter augmentation matrix of the input image; d s =1 / D s (u s ,v s ), O s (u s ,v s ) corresponding to the 3D point, remove the dimension where 1 is located, and keep O s The first three dimensions of O' s , using the PointsRasterizer class in the PyTorch3D library, O' s Project it to the target viewpoint image and render the z-buffer under the target viewpoint image. The z-buffer is the target viewpoint depth map D t .

[0043] In step S4, the projected image W s The calculation method is as follows:

[0044] First, use S3 to generate the target viewpoint depth map D t , the following formula is used to obtain the pixel of the target viewpoint image in the input viewpoint image I s The corresponding grid coordinates G t (u' t , v' t ):

[0045]

[0046] in, is the homogeneous coordinate of the pixel in the input viewpoint image, u' s is the input viewpoint image I s The coordinate of the pixel in the image coordinate system on the X axis, v' s is the input viewpoint image I s The coordinate of the pixel in the image coordinate system on the Y axis, d s =1 / D s (u' s ,v' s ), is the homogeneous coordinate of the pixel in the target viewpoint image, u' tis the coordinate of the pixel in the target viewpoint image on the X-axis of the image coordinate system, v' t is the coordinate of the pixel in the target viewpoint image on the Y axis of the image coordinate system, d t =1 / D t (u' t ,v' t ), P s is the camera extrinsic augmentation matrix of the input image, K s is the camera intrinsic parameter augmentation matrix of the input image, P t is the camera extrinsic augmentation matrix of the target viewpoint, is the inverse of the camera extrinsic augmentation matrix of the target viewpoint, K t is the camera intrinsic parameter augmentation matrix of the target viewpoint, is the inverse of the camera intrinsic parameter augmentation matrix of the target viewpoint;

[0047] Finally, G s with I s Input the grid_sample function of the PyTorch network together to get the transformed image W s ;

[0048] In step S4, the PSV generation method is specifically as follows:

[0049] In D t The maximum value of and minimum value According to the following formula, Y depth values ​​are taken. The value range of Y is 32≤Y≤256.

[0050]

[0051] Get Z d =[z1,z2,…,z Y ], calculate the grid coordinates G of PSV PSV (u” t ,v” t ):

[0052]

[0053] in, and They represent the homogeneous coordinates of different pixels of the input viewpoint image and the target viewpoint image projected at the same 3D point in the image coordinate system, and the homogeneous coordinates of the corresponding pixels of the input viewpoint image and the target viewpoint image in the image coordinate system, u' s ',v' s 'represents the coordinates of the X-axis and Y-axis of the image coordinate system corresponding to the pixel in the input viewpoint image, n is the normal vector of the plane, θ is the plane offset, n = [0, 0, 1], θ = -z i, R and T represent the relative poses between cameras.

[0054] In step S5, the fusion network is constructed by fusing a self-attention mechanism encoder and a soft attention mechanism decoder; the encoder comprises three modules, namely a swin-transformer network module, a convolutional neural network module with residual connection, and a feature modulation module;

[0055] Set the depth parameter of the swin-transformer network module to 4 and input W s , output transformer features through the swin-transformer network module Feature F μ The resolutions are in the following order: (W×H),

[0056] The convolutional neural network module with residual connection is a convolutional neural network module with a residual block in which the input skips the convolution layer and is directly added to the output. The convolutional neural network module contains four subnetworks, namely CNN1, CNN2, CNN3 and CNN4, among which CNN1 contains 2 residual blocks, CNN2 contains 2 residual blocks, CNN3 contains 6 residual blocks, and CNN4 contains 2 residual blocks; W s After inputting into the convolutional neural network module, first W s After two residual blocks in CNN1, we get Then After downsampling through the two residual blocks in CNN2, we get then After downsampling through the 6 residual blocks in CNN3, at last After downsampling through two residual blocks in CNN4, we get Final W s CNN features are obtained through the convolutional neural network module F v Resolution and F μ The resolution is the same;

[0057] Feature F v After the feature is obtained by the feature modulation module, the modulated feature F is obtained. m , the modulation process is as follows:

[0058]

[0059] Among them, IN represents instance normalization, J γ (·) and J β (·) indicates a convolutional layer whose parameters are not shared. Respectively represent element-wise multiplication and element-wise addition; F m Modulation features with four different resolutions

[0060] In step S5, the decoder includes two core modules, namely, a SKFF module (Selective Kernel Feature Fusion Module) and a DAU module (Dual Attention Unit Module);

[0061] The SKFF module accepts M features of shape H×W×X, 32≤M≤128. First, the M features input to the SKFF module are directly added to obtain the mixed feature B, and the global average pooling is calculated for B to obtain the channel feature F. B , F B The length of F is X. In this case, a convolutional layer is used to B Downsample and output the downsampled channel feature F' with a length of X / 8 B , and finally use M parallel convolutional layers to process F' B And calculate softmax to get the channel allocation weights of M upsampled channels Finally, Z F Multiply it with M features as the output of the SKFF module;

[0062] The DAU module accepts a feature F of size H×W×X' D , F D First, we get F' through two convolutional layers. D , F' D It is sent to two branches; in the first branch, F' is calculated in the feature channel S The global average pooling and global maximum pooling are performed to obtain two H×W×1 spatial features. These two spatial features are concatenated together to obtain a preliminary spatial feature S' of size H×W×2. S' is processed by a convolutional layer and then output to obtain the final spatial attention network. In the second branch, the spatial channel is calculated for F' D The global average pooling is performed to obtain the 1×1×X' preparatory channel features. The preparatory channel features are processed by two convolutional layers and then the channel attention is output. According to the propagation mechanism of the PyTorch network, F' D Respectively and Multiply to get the channel attention feature and Next, and After splicing, the output size is H×W×2X' combined features Will After a convolutional layer, D Add them together as the output of the DAU module;

[0063] In step S5, the new viewpoint is synthesized using the fusion network to obtain a preliminary image. The specific process is as follows:

[0064] First, the modulation feature F output by the fusion network is m After 3-5 convolutional layers, upsampling and downsampling are performed. By controlling the step size of the convolutional layer and the output channel parameters, the upsampling and downsampling ratios and the number of output channels can be controlled. During the sampling process, the convolutional layer parameters in all upsampling blocks and downsampling blocks are not shared; through sampling, three groups of resolutions are obtained, which are (H×W), Each set of features includes modulation features F m , input the three sets of features into three SKFF modules respectively, and after the output of the SKFF modules, input them into three DAU modules respectively to obtain the output features of the three DAU modules; sample the output features of the three DAU modules to a resolution of H×W, and then input them into a SKFF module; F m After the decoder, the preliminary image I of the fused new viewpoint under the target viewpoint of the video conference scene is obtained. * ;

[0065] In step S6, the process of obtaining the PSV fusion weight and opacity is as follows:

[0066] First, the PSV obtained by S4 and the preliminary image I of the new viewpoint are obtained by S5 * Splicing is performed at the channel level to obtain I in , the size of PSV is H×W×3(Y×V), V is the number of input viewpoints, I * The dimensions are H×W×3, I in The size is H×W×3(Y×V+1);

[0067] Then, I in The image is input into the U-Net network, and after passing through the U-Net network, the image fusion weight ω and opacity σ are output, where the size of ω is H×W×(Y×V) and the size of σ is H×W×(Y+1).

[0068] In step S7, the preliminary image I is fused using an alpha fusion algorithm. * Fusion is performed to obtain the target viewpoint image I t , the specific process is as follows:

[0069] Using the fusion weight ω and PSV obtained by S6, PSV is converted into the shape of H×W×3×Y×V. Based on the propagation mechanism of PyTorch, ω is directly multiplied by PSV. Using the sum function of Pytorch, the sum channel of the sum function is set to -1, and the output shape is H×W×3×Y PSV. ω ;

[0070] Finally, use PSV ω with I * Stitching, get PSV' of size H×W×3×(Y+1); alpha fusion of PSV' and σ, to get the final target viewpoint image I t .

[0071] An electronic device, comprising one or more processors; and a memory; and one or more programs, characterized in that the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to execute the new viewpoint synthesis method described in steps S1 to S7 above.

[0072] A computer-readable storage medium, characterized in that the computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the new viewpoint synthesis method described in steps S1 to S7 above.

[0073] The beneficial effects of the present invention are as follows: under large parallax conditions, sparse viewpoint images are used as input, and high-quality dense new viewpoint reconstruction of the video conferencing scene is performed through image processing technology and deep learning networks; the present invention does not require additional training for unseen scenes and has full generalization capabilities; the present invention achieves training acceleration through clear and geometrically meaningful mathematical models and algorithms, using a fully explicit expression method, while improving the adaptability and computational efficiency of the video conferencing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is an operation flow chart;

[0075] Figure 2 This is a diagram of the fusion network structure;

[0076] Figure 3 This is a visual comparison chart. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] The present invention proposes a new viewpoint synthesis method for video conferencing scenes. Under large parallax conditions, sparse viewpoint images are used as input, and high-quality dense new viewpoint reconstruction of video conferencing scenes is performed through image processing technology and deep learning networks. The present invention does not require additional training for unseen scenes and has full generalization capabilities; through clear and geometrically meaningful mathematical models and algorithms, training acceleration is achieved using a fully explicit expression method, while improving the adaptability and computational efficiency of the video conferencing system.

[0079] The process of synthesizing new viewpoints for video conferencing scenarios according to the present invention is as follows: Figure 1 , where input viewpoint 1 and input viewpoint 2 are represented by I1 and I2, steps S1 to S3 are depth estimation, and the calculation of projection result and PSV are represented by step S4, and PSV and projection result respectively represent W calculated in S4 s and PSV, the construction of the self-attention encoder and the soft attention decoder is step S5, the temporal feature fusion network is looser than steps S6 to S7, and the target viewpoint result is I t .

[0080] The specific steps of the new viewpoint synthesis method for video conferencing scenarios are as follows:

[0081] Step S1: Use a binocular depth estimation method to predict the input viewpoint image to obtain a depth map D' s ;

[0082] Step S2: Reduce the scene depth range to adjust the depth map D' s Perform fine correction to obtain the corrected depth map D s ;

[0083] Step S3: According to D s Calculate the target viewpoint depth map D t ;

[0084] Step S4: Calculate the projection image W of the input viewpoint image s With PSV;

[0085] Step S5: constructing a fusion network;

[0086] Step S6, obtaining the fusion weight and opacity of PSV;

[0087] Step S7: Obtain the target viewpoint image I using the alpha fusion algorithm t .

[0088] In step S1, the depth map D' s The prediction process is as follows:

[0089] Step S11, using a binocular camera Unimatch to capture two input viewpoint images, the binocular camera includes an input camera and a target camera, the two input viewpoint images are I1 and I2 respectively, and image features of I1 and I2 are extracted through a feature extraction network, the image features are F1 and F2 respectively; the feature extraction network uses a transformer network;

[0090] Step S12, through depth sequence transformation, obtain the image features of F2 under the corresponding camera posture of F1 The depth sequence transformation process is as follows:

[0091] First, the scene depth range is manually given, the given depth range is 200mm-5000mm, within the given depth range, the depth sequence is a i , i=1,2,…,N, the value range of N is 32≤N≤128; the grid coordinates are calculated according to the depth sequence, and the grid coordinate transformation process is:

[0092]

[0093] Where H(·) represents the transformation of homogeneous grid coordinates, is the grid coordinate of F1 corresponding to the camera pose, G 2d is the grid coordinate of F2 corresponding to the camera pose, K1 represents the intrinsic parameter matrix of I1 corresponding to the camera, K2 represents the intrinsic parameter matrix of I2 corresponding to the camera, E1 represents the extrinsic parameter matrix of I1 corresponding to the camera, E2 represents the extrinsic parameter matrix of I2 corresponding to the camera, H and W represent the height and width of F2;

[0094] Then, normalize Decompose H×W×2 into two H×W×1 shapes and Will Divide by W, and we get Will Dividing by H, we get Then re- and Splice into a grid coordinate with a shape of H×W×2 The normalized

[0095]

[0096] Finally, the normalized Input the grid_sample function of the PyTorch network together with F1 to output image features

[0097] Step S13, calculating the image feature correlation C i ; The image feature correlation C i The calculation formula is as follows:

[0098]

[0099] Where D is the dimension of the feature number channel of F1;

[0100] Step S14: according to the image feature correlation C i Calculate the depth map D' s ; The depth map D' s The calculation process is as follows:

[0101] First, C i Set, get C d =[C1,C2,…,C N ];

[0102] Then, C d After normalization, we get M d , the normalized formula is: M d = SOFTMAX(C d );

[0103] Finally, PyTorch’s propagation mechanism is used to propagate M d With A d Multiply them to get the depth map D' of the input viewpoint image s , where s represents different input viewpoint images, A d for a i The depth sequence A d =[a1,a2,…,a N ].

[0104] In step S2, the fine correction process is as follows:

[0105] Step S21, narrow the depth range described in step S12 to 200mm-500mm, and repeat the above steps S12 to S14; during the process, the depth sequence diagram b i The collection is B d =[b1,b2,…,b N ], the transformed image features are

[0106] Step S22, calculate N The variance Q with F1 d , Q d Input into the U-Net network to get the depth probability P d , according to formula D s =B d ·ρ d , get the corrected input viewpoint image depth map D s .

[0107] In step S3, the target viewpoint depth map D t The calculation method is as follows:

[0108] Since the depth map D s Contains the corresponding depth of each pixel in the input viewpoint image, D s (u s ,v s ) After back-projection, we get the 3D point cloud O s (u s ,v s ), back projection calculation formula:

[0109]

[0110] in, is the inverse of the camera extrinsic augmented matrix of the input image; The inverse of the camera intrinsic parameter augmentation matrix of the input image; d s =1 / D s (u s ,v s ), O s (u s ,v s ) corresponding to the 3D point, remove the dimension where 1 is located, and keep O s The first three dimensions of O' s , using the PointsRasterizer class in the PyTorch3D library, O' s Project it to the target viewpoint image and render the z-buffer under the target viewpoint image. The z-buffer is the target viewpoint depth map D t .

[0111] In step S4, the projected image W s The calculation method is as follows:

[0112] First, use S3 to generate the target viewpoint depth map D t , the following formula is used to obtain the pixel of the target viewpoint image in the input viewpoint image I s The corresponding grid coordinates G t (u' t, v' t ):

[0113]

[0114] in, is the homogeneous coordinate of the pixel in the input viewpoint image, u' s is the input viewpoint image I s The coordinate of the pixel in the image coordinate system on the X axis, v' s is the input viewpoint image I s The coordinate of the pixel in the image coordinate system on the Y axis, d s =1 / D s (u' s ,v' s ), is the homogeneous coordinate of the pixel in the target viewpoint image, u' t is the coordinate of the pixel in the target viewpoint image on the X-axis of the image coordinate system, v' t is the coordinate of the pixel in the target viewpoint image on the Y axis of the image coordinate system, d t =1 / D t (u' t ,v' t ), P s is the camera extrinsic augmentation matrix of the input image, K s is the camera intrinsic parameter augmentation matrix of the input image, P t is the camera extrinsic augmentation matrix of the target viewpoint, is the inverse of the camera extrinsic augmentation matrix of the target viewpoint, K t is the camera intrinsic parameter augmentation matrix of the target viewpoint, is the inverse of the camera intrinsic parameter augmentation matrix of the target viewpoint;

[0115] Finally, G s with I s Input the grid_sample function of the PyTorch network together to get the transformed image W s .

[0116] In step S4, the PSV generation method is specifically as follows:

[0117] In D t The maximum value of and minimum value According to the following formula, Y depth values ​​are taken. The value range of Y is 32≤Y≤256.

[0118]

[0119] Get Z d =[z1,z2,…,z Y], calculate the grid coordinates G of the PSV (Plane Sweep Volume) PSV (u” t ,v” t ):

[0120]

[0121] in, and They represent the homogeneous coordinates of different pixels of the input viewpoint image and the target viewpoint image projected at the same 3D point in the image coordinate system, and the homogeneous coordinates of the corresponding pixels of the input viewpoint image and the target viewpoint image in the image coordinate system, u' s ',v' s 'represents the coordinates of the X-axis and Y-axis of the image coordinate system corresponding to the pixel in the input viewpoint image, n is the normal vector of the plane, θ is the plane offset, n = [0, 0, 1], θ = -z i , R and T represent the relative poses between cameras.

[0122] In step S5, the fusion network is constructed by fusing the self-attention mechanism encoder and the soft attention mechanism decoder; the encoder includes three modules, namely, a swin-transformer network module, a convolutional neural network module with residual connection, and a feature modulation module; the fusion network structure is as follows Figure 2 As shown, SW-E represents a swin-transformer encoder, SW-E1 represents a swin-transformer encoder with a depth of 1, SW-E2 represents a swin-transformer encoder with a depth of 2, SW-E3 represents a swin-transformer encoder with a depth of 3, and SW-E4 represents a swin-transformer encoder with a depth of 4. CNN-E represents a CNN encoder, CNN-E1 represents a CNN encoder with a depth of 1, CNN-E2 represents a CNN encoder with a depth of 2, CNN-E3 represents a CNN encoder with a depth of 3, and CNN-E4 represents a CNN encoder with a depth of 4; modulation represents a feature modulation module, modulation1 to modulation4 represent four feature modulation modules that do not share parameters, and MF represents the output modulation feature F m MF1 to MF4 represent SKFF and DAU represent the SKFF module and DAU module respectively.

[0123] Set the depth parameter of the swin-transformer network module to 4 and input W s, output transformer features through the swin-transformer network module Feature F μ The resolutions are in the following order: (W×H),

[0124] The convolutional neural network module with residual connection is a convolutional neural network module with a residual block in which the input skips the convolution layer and is directly added to the output. The convolutional neural network module contains four subnetworks, namely CNN1, CNN2, CNN3 and CNN4, among which CNN1 contains 2 residual blocks, CNN2 contains 2 residual blocks, CNN3 contains 6 residual blocks, and CNN4 contains 2 residual blocks; W s After inputting into the convolutional neural network module, first W s After two residual blocks in CNN1, we get Then After downsampling through the two residual blocks in CNN2, we get then After downsampling through the 6 residual blocks in CNN3, at last After downsampling through two residual blocks in CNN4, we get Final W s CNN features are obtained through the convolutional neural network module F v Resolution and F μ The resolution is the same;

[0125] Feature F v After the feature is obtained by the feature modulation module, the modulated feature F is obtained. m , the modulation process is as follows:

[0126]

[0127] Among them, IN represents instance normalization, J γ (·) and J β (·) indicates a convolutional layer whose parameters are not shared. Respectively represent element-wise multiplication and element-wise addition; F m Modulation features with four different resolutions

[0128] In step S5, the decoder includes two core modules, namely, a SKFF module (Selective Kernel Feature Fusion Module) and a DAU module (Dual Attention Unit Module);

[0129] The SKFF module accepts M features of shape H×W×X, 32≤M≤128. First, the M features input to the SKFF module are directly added to obtain the mixed feature B, and the global average pooling is calculated for B to obtain the channel feature F. B , F B The length of FB is X. At this time, a convolutional layer is used to downsample FB and output the downsampled channel feature F' with a length of X / 8. B , and finally use M parallel convolutional layers to process F' B And calculate softmax to get the channel allocation weights of M upsampled channels Finally, Z F Multiply it with M features as the output of the SKFF module;

[0130] The DAU module accepts a feature F of size H×W×X' D , F D First, we get F' through two convolutional layers. D , F' D It is sent to two branches; in the first branch, F' is calculated in the feature channel D The global average pooling and global maximum pooling are performed to obtain two H×W×1 spatial features. These two spatial features are concatenated together to obtain a preliminary spatial feature S' of size H×W×2. S' is processed by a convolutional layer and then output to obtain the final spatial attention network. In the second branch, the spatial channel is calculated for F' D The global average pooling is performed to obtain the 1×1×X' preparatory channel features. The preparatory channel features are processed by two convolutional layers and then the channel attention is output. According to the propagation mechanism of the PyTorch network, F' D Respectively and Multiply to get the channel attention feature and Next, and After splicing, the output size is H×W×2X' combined features Will After a convolutional layer, D Add them together as the output of the DAU module;

[0131] In step S5, the new viewpoint is synthesized using the fusion network to obtain a preliminary image. The specific process is as follows:

[0132] First, the modulation feature F output by the fusion network is mAfter 3-5 convolutional layers, upsampling and downsampling are performed. By controlling the step size of the convolutional layer and the output channel parameters, the upsampling and downsampling ratios and the number of output channels can be controlled. During the sampling process, the convolutional layer parameters in all upsampling blocks and downsampling blocks are not shared; through sampling, three groups of resolutions are obtained, which are (H×W), Each set of features includes modulation features F m , input the three sets of features into three SKFF modules respectively, and after the output of the SKFF modules, input them into three DAU modules respectively to obtain the output features of the three DAU modules; sample the output features of the three DAU modules to a resolution of H×W, and then input them into a SKFF module; F m After the decoder, the preliminary image I of the fused new viewpoint under the target viewpoint of the video conference scene is obtained. * .

[0133] In step S6, the process of obtaining the PSV fusion weight and opacity is as follows:

[0134] First, the PSV obtained by S4 and the preliminary image I of the new viewpoint are obtained by S5 * Splicing is performed at the channel level to obtain I in , the size of PSV is H×W×3(Y×V), V is the number of input viewpoints, I * The dimensions are H×W×3, I in The size is H×W×3(Y×V+1);

[0135] Then, I in The image is input into the U-Net network, and after passing through the U-Net network, the image fusion weight ω and opacity σ are output, where the size of ω is H×W×(Y×V) and the size of σ is H×W×(Y+1).

[0136] In step S7, the preliminary image I is fused using an alpha fusion algorithm. * Fusion is performed to obtain the target viewpoint image I t , the specific process is as follows:

[0137] Using the fusion weight ω and PSV obtained by S6, PSV is converted into the shape of H×W×3×Y×V. Based on the propagation mechanism of PyTorch, ω is directly multiplied by PSV. Using the sum function of Pytorch, the sum channel of the sum function is set to -1, and the output shape is H×W×3×Y PSV. ω ;

[0138] Finally, use PSV ω with I *Stitching, get PSV' of size H×W×3×(Y+1); alpha fusion of PSV' and σ, to get the final target viewpoint image I t .

[0139] An electronic device, comprising one or more processors; and a memory; and one or more programs, characterized in that the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to execute the new viewpoint synthesis method described in steps S1 to S7 above.

[0140] A computer-readable storage medium, characterized in that the computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the new viewpoint synthesis method described in steps S1 to S7 above.

[0141] Experimental comparison:

[0142] The present invention obtains high-quality new view synthesis results at the target viewpoint and achieves the best results in comparison with other methods. The visual effect comparison after visualization is shown in Figure 2. Figure 3 . Among them, the comparison methods are: 1) ENeRF, the best method for synthesizing new views with short baselines (small distance between cameras) in 2022, 2) AttnRend, the best method for synthesizing new views with wide baselines (large distance between cameras) in 2023, 3) MuRF, the best method for synthesizing new views with multiple baselines (applicable to different camera spacings) based on implicit expression in 2024, and 4) MVSplat, the best method for synthesizing new views based on 3D Gaussian in 2024. The result of the present invention is Ours, and GT represents the true value. This method is superior to other comparison methods in multiple evaluation indicators, including but not limited to peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceptual image quality index (LPIPS). The specific results are shown in Table 1, where "w mask" means that only the quantitative evaluation index of the human body area is calculated, and "w / o mask" means that the quantitative evaluation index of the human body and the background is calculated simultaneously. The symbol "↑" means that the larger the value, the better the effect, and "↓" means that the smaller the value, the better the effect, and the best result is marked in bold. To protect the portrait rights of the characters, the visualization results only retain the facial area with artifacts, and the rest of the parts are mosaic processed. These results intuitively demonstrate the superiority of this method in processing complex human scenes, especially the significant improvement in reducing facial artifacts.

[0143] Table 1

[0144]

[0145] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A new viewpoint synthesis method for video conferencing scenarios, characterized in that: The new viewpoint synthesis method comprises the following steps: Step S1: Use a binocular depth estimation method to predict the input viewpoint image to obtain a depth map D′ s ; Step S2: Reduce the scene depth range to obtain the depth map D′ s Perform fine correction to obtain the corrected depth map D s ; Step S3: According to D s Calculate the target viewpoint depth map D t ; Step S4: Calculate the projection image W of the input viewpoint image s With PSV; Step S5: constructing a fusion network; Step S6, obtaining the fusion weight and opacity of PSV; Step S7: Obtain the target viewpoint image I using the alpha fusion algorithm t .

2. A new viewpoint synthesis method for video conferencing scenes according to claim 1, characterized in that: In step S1, the depth map D′ s The prediction process is as follows: Step S11, a binocular camera is used to capture two input viewpoint images, the binocular camera includes an input camera and a target camera, the two input viewpoint images are O1 and I2 respectively, image features of I1 and I2 are extracted through a feature extraction network, and the image features are F1 and F2 respectively; the feature extraction network uses a transformer network; Step S12, through depth sequence transformation, obtain the image features of F2 under the corresponding camera posture of F1 The depth sequence transformation process is as follows: First, manually specify the scene depth range. The given depth range is 200mm - 5000mm. Within the given depth range, the depth sequence is a i , where i = 1, 2,..., N, and the value range of N is 32 < N < 128; calculate the grid coordinates according to the depth sequence. The grid coordinate transformation process is as follows: Where H(·) represents the transformation of homogeneous grid coordinates, is the grid coordinate of F1 under the camera pose, G2d is the grid coordinate of F2 under the camera pose, K1 represents the intrinsic parameter matrix of I1, K2 represents the intrinsic parameter matrix of I2, E1 represents the extrinsic parameter matrix of I1, E2 represents the extrinsic parameter matrix of I2, H and W represent the height and width of F2; Then, normalize Decompose H×W×2 into two H×W×1 shapes and Will Divide by W, and we get Will Divide by H, and we get Then re- and Splice into a grid coordinate with a shape of H×W×2 The normalized Finally, the normalized Input the grid_sample function of the PyTorch network together with F1 to output image features Step S13, calculating the image feature correlation C i ; The image feature correlation C i The calculation formula is as follows: Where D is the dimension of the feature number channel of F1; Step S14: according to the image feature correlation C i Calculate the depth map D′ s ; The depth map D′ s The calculation process is as follows: First, C i Set, get C d =[C1,C2,...,C N ]; Then, Cd is normalized to obtain M d , the normalized formula is: M d = SOFTMAX(C d ); Finally, PyTorch’s propagation mechanism is used to propagate M d With A d Multiply them to get the depth map D′ of the input viewpoint image s , where s represents different input viewpoint images, A d for a i The depth sequence A d =[a1, a2, ..., a N ].

3. A new viewpoint synthesis method for video conferencing scenes according to claim 2, characterized in that: In step S2, the fine correction process is as follows: Step S21, narrow the depth range of step S12 to 200mm-500mm, and repeat steps S12 to S14; during the process, the depth sequence diagram b i The collection is B d =[b1, b2, ..., b N ], the transformed image features are Step S22, calculate N The variance Q with F1 d , Q d Input into the U-Net network to get the depth probability P d , according to formula D s =B d ·ρ d , get the corrected input viewpoint image depth map D s .

4. A new viewpoint synthesis method for video conferencing scenes according to claim 1, characterized in that: In step S3, the target viewpoint depth map D t The calculation method is as follows: Since the depth map D s Contains the corresponding depth of each pixel in the input viewpoint image, D s (u s , v s ) After back-projection, we get the 3D point cloud O s (u s , v s ), back projection calculation formula: in, is the inverse of the camera extrinsic augmented matrix of the input image; The inverse of the camera intrinsic parameter augmentation matrix of the input image; d s =1 / D s (u s , v s ), O s (u s , v s ) corresponding to the 3D point, remove the dimension where 1 is located, and keep O s The first three dimensions of O' s , using the PointsRasterizer class in the PyTorch3D library, O' s Project it to the target viewpoint image and render the z-buffer under the target viewpoint image. The z-buffer is the target viewpoint depth map D t .

5. The new viewpoint synthesis method for video conferencing scenes according to claim 1, characterized in that: In step S4, the projected image W s The calculation method is as follows: First, use S3 to generate the target viewpoint depth map D t , the following formula is used to obtain the pixel of the target viewpoint image in the input viewpoint image I s The corresponding grid coordinates G t (u′ t , v′ t ): in, is the homogeneous coordinate of the pixel in the input viewpoint image, u′ s is the input viewpoint image I s The coordinate of the pixel in the image coordinate system on the X axis, v′ s is the input viewpoint image I s The coordinate of the pixel in the image coordinate system on the Y axis, d s =1 / D s (u' s , v' s ), is the homogeneous coordinate of the pixel in the target viewpoint image, u′ t is the coordinate of the pixel in the target viewpoint image on the X-axis of the image coordinate system, v′ t is the coordinate of the pixel in the target viewpoint image on the Y axis of the image coordinate system, d t =1 / D t (u' t , v' t ), P s is the camera extrinsic augmentation matrix of the input image, K s is the camera intrinsic parameter augmentation matrix of the input image, P t is the camera extrinsic augmentation matrix of the target viewpoint, is the inverse of the camera extrinsic augmentation matrix of the target viewpoint, K t is the camera intrinsic parameter augmentation matrix of the target viewpoint, is the inverse of the camera intrinsic parameter augmentation matrix of the target viewpoint; Finally, G s with I s Input the grid_sample function of the PyTorch network together to get the transformed image W s ; In step S4, the PSV generation method is specifically as follows: In D t The maximum value of and minimum value According to the following formula, Y depth values ​​are taken. The value range of Y is 32≤Y≤256. Get Z d =[z1,z2,...,z Y ], calculate the grid coordinates G of the PSV (Plane Sweep Volume) PSV (u″ t ,v″ t ): in, and They represent the homogeneous coordinates of different pixels of the input viewpoint image and the target viewpoint image projected at the same 3D point in the image coordinate system, and the homogeneous coordinates of the corresponding pixels of the input viewpoint image and the target viewpoint image in the image coordinate system, u″ s ,v″ s They represent the X-axis and Y-axis coordinates of the pixel in the input viewpoint image, n is the normal vector of the plane, θ is the plane offset, n = [0, 0, 1], θ = -z i , R and T represent the relative poses between cameras.

6. A new viewpoint synthesis method for video conferencing scenes according to claim 1, characterized in that: In step S5, the fusion network is constructed by fusing a self-attention mechanism encoder and a soft attention mechanism decoder; the encoder comprises three modules, namely a swin-transformer network module, a convolutional neural network module with residual connection, and a feature modulation module; Set the depth parameter of the swin-transformer network module to 4 and input W s , output transformer features through the swin-transformer network module Feature F μ The resolutions are in the order of (W×H), The convolutional neural network module with residual connection is a convolutional neural network module with a residual block in which the input skips the convolution layer and is directly added to the output. The convolutional neural network module contains four sub-networks, namely CNN1, CNN2, CNN3 and CNN4, among which CNN1 contains 2 residual blocks, CNN2 contains 2 residual blocks, CNN3 contains 6 residual blocks, and CNN4 contains 2 residual blocks; W s After inputting into the convolutional neural network module, first W s After two residual blocks in CNN1, we get Then After downsampling through the two residual blocks in CNN2, we get then After downsampling through the 6 residual blocks in CNN3, at last After downsampling through two residual blocks in CNN4, we get Final W s CNN features are obtained through the convolutional neural network module F v Resolution and F μ The resolution is the same; The feature Fv is modulated by the feature modulation module to obtain the modulated feature F m , the modulation process is as follows: Among them, IN represents instance normalization, J γ (·) and J β (·) indicates a convolutional layer whose parameters are not shared. Respectively represent element-wise multiplication and element-wise addition; F m Modulation features with four different resolutions In step S5, the decoder includes two core modules, namely, a SKFF module (Selective Kernel Feature Fusion Module) and a DAU module (Dual Attention Unit Module); The SKFF module accepts M features each with a shape of H×W×X, where 32 < M < 128. First, the M features input to the SKFF module are directly added together to obtain a mixed feature B. Then, global average pooling is calculated on B to obtain a channel feature F B , F B has a length of X. At this time, a convolutional layer is used to downsample F B to output a downsampled channel feature F′ with a length of X / 8 B . Finally, M parallel convolutional layers are used to process F′ B and softmax is calculated to obtain M upsampled channel allocation weights Finally, multiply Z F by the M features as the output of the SKFF module; The DAU module accepts a feature F of size H×W×X′ D , F D First, we get F′ through two convolutional layers. D , F′ D It is sent to two branches; in the first branch, F′ is calculated in the feature channel D The global average pooling and global maximum pooling are performed to obtain two H×W×1 spatial features. These two spatial features are concatenated together to obtain a preliminary spatial feature S′ of size H×W×2. S′ is processed by a convolutional layer and then output to obtain the final spatial attention network. In the second branch, the pair F′ is calculated in the spatial channel D The global average pooling is performed to obtain the 1×1×X′ preparatory channel features. The preparatory channel features are processed by two convolutional layers and then the channel attention is output. According to the propagation mechanism of the PyTorch network, F′ is directly D Respectively and Multiply to get the channel attention feature and Next, and After splicing, the output size is H×W×2X′ combined features Will After a convolutional layer, D Add them together as the output of the DAU module; In step S5, the new viewpoint is synthesized using the fusion network to obtain a preliminary image. The specific process is as follows: First, the modulation feature F output by the fusion network is m After 3-5 convolutional layers, upsampling and downsampling are performed. By controlling the step size of the convolutional layer and the output channel parameters, the upsampling and downsampling ratios and the number of output channels can be controlled. During the sampling process, the convolutional layer parameters in all upsampling blocks and downsampling blocks are not shared; through sampling, three groups of resolutions are obtained, which are (H×W) in order. Each set of features includes modulation features F m , input the three sets of features into three SKFF modules respectively, and after the output of the SKFF modules, input them into three DAU modules respectively to obtain the output features of the three DAU modules; sample the output features of the three DAU modules to a resolution of H×W, and then input them into a SKFF module; F m After the decoder, the preliminary image I* of the new viewpoint after fusion under the target viewpoint of the video conference scene is obtained.

7. A new viewpoint synthesis method for video conferencing scenes according to claim 1, characterized in that: In step S6, the process of obtaining the PSV fusion weight and opacity is as follows: First, the PSV obtained by S4 and the preliminary image I of the new viewpoint are obtained by S5 * Splicing is performed at the channel level to obtain I in , the size of PSV is H×W×3(Y×V), V is the number of input viewpoints, I * The dimensions are H×W×3, I in The size is H×W×3(Y×V+1); Then, I in The image is input into the U-Net network, and after passing through the U-Net network, the image fusion weight ω and opacity σ are output, where the size of ω is H×W×(Y×V) and the size of σ is H×W×(Y+1).

8. The new viewpoint synthesis method for video conferencing scenes according to claim 1, characterized in that: In step S7, the preliminary image I is fused using an alpha fusion algorithm. * Fusion is performed to obtain the target viewpoint image I t , the specific process is as follows: Using the fusion weight ω and PSV obtained by S6, PSV is converted into the shape of H×W×3×Y×V. Based on the propagation mechanism of PyTorch, ω and PSV are directly multiplied. Using the sum function of Pytorch, the sum channel of the sum function is set to -1, and the output shape is H×W×3×Y PSV. ω ; Finally, use PSV ω with I * Stitching, get PSV′ of size H×W×3×(Y+1); alpha fusion of PSV′ and σ, to get the final target viewpoint image I t .

9. An electronic device comprising one or more processors; and a memory; and one or more programs, characterized in that: The one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 8.

Images