Plane video three-dimensional conversion system and method, computer equipment and storage medium

Abstract

The invention discloses a planar video three-dimensional conversion system and method, computer equipment and a storage medium, and belongs to the technical field of computer vision. The method comprises the following steps: extracting coarse-grained depth features of video frames in a plane video through a depth sensing module; extracting spatial detail features of video frames in the plane video through a spatial detail module; fusing the coarse-grained depth features and the spatial detail features through a depth enhancement module to generate fine-grained depth features, and decoding the fine-grained depth features into a planar layered mask; multiplying the spatial detail features by the plane layered mask through a multi-plane generation module to obtain multi-plane image features, and decoding the multi-plane image features to obtain multi-plane images; and forming a stereoscopic video frame through the multiple plane images, and finally outputting the stereoscopic video frame. Therefore, the three-dimensional expression is constructed by using the multi-level image, the three-dimensional planar video can be quickly and accurately realized, the robustness is high, and the generalization is strong.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a system and method for converting a two-dimensional video into a three-dimensional video, a computer device and a storage medium. Background Art

[0002] Traditional 3D filmmaking usually relies on using two or more special cameras to shoot the same scene simultaneously. This method simulates human binocular stereoscopic vision by capturing the same scene from different angles, and then presenting the two perspectives to the audience's left and right eyes through 3D glasses or head-mounted devices during playback, thereby creating a sense of depth. However, this shooting method is very expensive, with high costs in terms of equipment investment, post-processing and production time. Therefore, this 3D shooting technology is usually only used in some high-quality film productions, while some daily videos, such as news reports, documentaries, educational videos, etc., are still shot in 2D due to cost considerations.

[0003] In recent years, display technology has developed rapidly, and various naked-eye 3D screens have emerged. These naked-eye 3D screens can provide viewers with a three-dimensional visual experience without the use of special 3D glasses. However, the corresponding 3D video resources are still in short supply. At present, only a small number of 3D videos on the market can be played on such naked-eye 3D screens, which to a certain extent limits the popularity and application of naked-eye 3D technology. Some current methods of stereoscopicization of flat videos usually require the assistance of depth maps. A depth map is an image in which the value of each pixel represents the distance between the corresponding scene point and the camera. The depth map can be obtained by manual drawing or algorithm estimation. However, whether it is manual drawing or algorithm estimation, the cost of stereoscopicization of flat videos will become high, and the real-time effect of stereoscopicization of flat videos will be poor.

[0004] After searching, the Chinese patent application, application number 201210209541.6, published on October 10, 2012, discloses a method for converting a flat video into a stereoscopic video and a system using the method. The conversion method includes the following steps: inputting a flat video; lens classification, selecting key frames; depth calculation, obtaining a key frame depth map; performing depth propagation, obtaining a depth map sequence; image rendering, obtaining a stereo image sequence; post-processing, obtaining the final stereo video. The flat video stereo conversion method of the invention uses a human-computer interaction method to input information and timely adjust algorithm parameters and call the corresponding conversion algorithm, which can intuitively and efficiently control the entire stereo video production process, improve the efficiency of depth map production and the accuracy of depth map generation. However, the scheme does not take into account the input of information, adjustment of algorithm parameters and calling of corresponding conversion algorithms through human-computer interaction, which requires a large manpower and time cost, and the conversion efficiency is not high. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] The present invention provides a planar video stereo conversion system, method, computer device and storage medium, which solve the problem that traditional stereo algorithms need depth map assistance and are difficult to achieve real-time processing through the generation and representation of lightweight multi-level images.

[0007] 2. Technical solution

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] A two-dimensional video stereo conversion system, comprising:

[0010] Input module, obtains and inputs planar video;

[0011] Depth perception module, extracts coarse-grained depth features of video frames in planar videos;

[0012] The spatial detail module extracts the spatial detail features of video frames in the planar video;

[0013] The depth enhancement module fuses the coarse-grained depth features and the spatial detail features to generate fine-grained depth features, and decodes the fine-grained depth features into planar hierarchical masks;

[0014] The multi-plane generation module multiplies the spatial detail features and the plane layered mask to obtain the multi-plane image features, and obtains the multi-plane image by decoding the multi-plane image features;

[0015] The output module forms a stereoscopic video frame through multi-plane images and outputs the stereoscopic video frame.

[0016] Furthermore, the depth perception module includes a feature extraction layer and a basic block unit; the feature extraction layer decomposes the video frame into a feature map, and the basic block unit extracts features from the feature map to obtain a coarse-grained depth feature; the coarse-grained depth feature is expressed as:

[0017]

[0018] Among them, d c represents the coarse-grained depth feature, I represents the video frame; F depth Represents the depth perception module.

[0019] Furthermore, the spatial detail module includes a spatial feature extraction block, which extracts spatial detail features in the video frame; the spatial detail features are expressed as:

[0020]

[0021] Among them, f d Indicates that F detail Represents the spatial detail module.

[0022] Furthermore, the depth enhancement module includes an attention extraction block, a feature fusion block and a mask decoding block; the attention extraction block extracts attention features from coarse-grained depth features, and the feature fusion block fuses attention features and spatial detail features to obtain fine-grained depth features; the fine-grained depth features are expressed as:

[0023]

[0024] Among them, d f represents the fine-grained deep features, F fusion Represents the depth enhancement module.

[0025] Furthermore, the mask decoding block decodes the fine-grained depth features into a planar hierarchical mask; the planar hierarchical mask is represented as:

[0026]

[0027] Where i represents a natural number, N represents the number of plane layers, and M i Represents a planar layering mask.

[0028] Furthermore, the multi-plane generation module includes an upsampling unit; the spatial detail features and the plane layered mask are fused to generate multi-plane image features, and the multi-plane image features are decoded by the upsampling unit to obtain a multi-plane image; the multi-plane image is represented as:

[0029]

[0030] Among them, P i represents a multi-plane image, F RGBα Represents the multi-plane generation module.

[0031] Furthermore, the multi-plane image is subjected to a plane transformation by using a homography transformation formula; the homography transformation formula is expressed as:

[0032]

[0033] Among them, x s Indicates the horizontal coordinate of the source perspective pixel, y s represents the vertical coordinate of the pixel point of the source perspective, t represents the translation vector between perspectives, n represents the plane normal vector, z n Indicates the plane depth value, x t Indicates the horizontal coordinate of the target viewing angle pixel, y t Represents the ordinate of the pixel point of the target viewing angle, T represents the transposition operation, K represents the camera intrinsic parameter, and R represents the rotation matrix between viewing angles.

[0034] A method for converting a two-dimensional video into a three-dimensional video comprises the following steps:

[0035] Get and input the flat video;

[0036] Extracting coarse-grained deep features of video frames in planar videos;

[0037] Extract spatial detail features of video frames in planar video;

[0038] Fusing coarse-grained depth features and spatial detail features to generate fine-grained depth features, and decoding the fine-grained depth features into planar hierarchical masks;

[0039] The multi-plane image features are obtained by fusing the spatial detail features and the plane layered mask, and the multi-plane image is obtained by decoding the multi-plane image features;

[0040] A stereoscopic video frame is formed by using multi-plane images, and the stereoscopic video frame is output.

[0041] A computer device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned method when executing the computer program.

[0042] A computer-readable storage medium stores a computer program, and the computer program executes the above method when executed by a processor.

[0043] 3. Beneficial effects

[0044] Compared with the prior art, the advantages of the present invention are:

[0045] The present invention provides a two-dimensional video stereo conversion system, method, computer device and storage medium. A lightweight depth perception module is constructed to obtain depth perception, thereby saving time and labor costs. The spatial detail features extracted by constructing a spatial detail module can not only enhance the depth perception information, but also be used to generate multi-plane image features. The depth enhancement module is constructed to enhance the depth information using detail features to obtain accurate depth perception information and local details. A multi-plane generation module is constructed to generate a multi-level image representation to achieve real-time two-dimensional video stereo conversion. The three-dimensional expression is constructed using multi-level images to quickly and accurately achieve two-dimensional video stereo conversion, with high robustness and strong generalization, and strong practicality and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the structure of a two-dimensional video stereo conversion system according to an embodiment of the present invention;

[0047] Figure 2This is a schematic diagram of the structure of a depth perception module according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of a space detail module according to an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of a depth enhancement module according to an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the structure of a multi-plane generation module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0052] Example

[0053] like Figure 1 As shown, a two-dimensional video stereo conversion system provided by this embodiment includes an input module, a depth perception module, a spatial detail module, a depth enhancement module, a multi-plane generation module and an output module.

[0054] Specifically in this embodiment, the input module is used to obtain and input a planar video.

[0055] like Figure 2As shown, the depth perception module is used to extract the coarse-grained depth features of the video frames in the planar video. In this embodiment, the depth perception module includes a feature extraction layer and a basic block unit. The feature extraction layer decomposes the video frames in the planar video into feature maps, and the basic block unit extracts features from the feature maps to obtain coarse-grained depth features. Specifically, in this embodiment, the depth perception module consists of a feature extraction layer and five basic unit blocks. The feature extraction layer consists of a 3×3 convolution layer, and the video frame is decomposed into a feature map of c channels through the feature extraction layer. Thus, the image in the video frame is converted into a high-dimensional representation through feature extraction to better capture important information in the image. In this embodiment, preferably, the video frame is decomposed into a feature map of 64 channels. In this embodiment, the structure of each basic block unit block includes: m convolution modules, a convolution layer with a convolution kernel size of 3×3, a batch normalization layer, and a Relu activation function layer. The convolution layer step size of the first convolution module in each basic unit block is 2, which is used to reduce the feature resolution length and width of the feature map to one-half, and the convolution layer step size of the remaining volume modules is 1. After the feature map passes through five basic block units, its feature resolution length and width are reduced to one thirty-second, thereby obtaining a coarse-grained depth feature. It should be noted that as the feature resolution decreases, the amount of calculation of the basic block unit will also decrease. Therefore, in this embodiment, the number of convolution modules of the first three basic block units is set to 2, and the number of convolution modules of the last two basic block units is set to 3, thereby improving the feature extraction accuracy.

[0056] In this embodiment, the coarse-grained depth feature of the video frame in the planar video is extracted by the depth perception module, and the coarse-grained depth feature is expressed as:

[0057]

[0058] Among them, d c represents the coarse-grained depth feature, I represents the video frame; F depth Represents the depth perception module.

[0059] Therefore, compared with the traditional stereo video conversion method that requires manual drawing or algorithm estimation of depth maps, thereby increasing additional time and labor costs, the depth perception module constructed in this embodiment is lightweight and fast, and the use of a fast downsampling strategy can increase the network receptive field, effectively extract coarse-grained depth features of video frames, and obtain depth perception information.

[0060] like Figure 3As shown, the spatial detail module is used to extract the spatial detail features of the video frame in the planar video. The spatial detail module includes a spatial feature extraction block, which extracts the spatial detail features in the video frame. In this embodiment, the spatial detail module includes three spatial feature extraction blocks, and the video frame is encoded by the three spatial feature extraction blocks. Each spatial feature extraction block extracts the detail features of the video frame and reduces the length and width of the image feature resolution in the video frame by half, and finally obtains the spatial detail features of one eighth of the video frame image resolution. Specifically, the first spatial feature extraction block includes: a continuous expansion convolution module consisting of 3 convolution kernels of size 3×3, step length 2, expansion rates of 1, 2, and 3 respectively, a batch normalization layer, and a GELU activation function layer; the second spatial feature extraction block includes: a continuous expansion convolution module consisting of 3 convolution kernels of size 3×3, step length 2, expansion rates of 1, 2, and 3 respectively, a batch normalization layer, and a GELU activation function layer; the third spatial feature extraction block includes: a continuous expansion convolution module consisting of 9 convolution kernels of size 3×3, step length 2, expansion rates of 1, 2, 3, 1, 2, 3, 1, 2, 3 respectively, a batch normalization layer, and a GELU activation function layer. It should be noted that as the resolution decreases, the amount of calculation of the spatial feature extraction block will also decrease. Therefore, in this embodiment, the number of convolution layers of the first spatial feature extraction block is set to 3, the number of convolution layers of the second spatial feature extraction block is set to 3, and the number of convolution layers of the third spatial feature extraction block is set to 9, thereby improving the feature extraction accuracy.

[0061] In this embodiment, the spatial detail features of the video frame in the planar video are extracted by the spatial detail module, and the spatial detail features are expressed as:

[0062]

[0063] Among them, f d Indicates that F detail Represents the spatial detail module.

[0064] Therefore, in this embodiment, the spatial detail features extracted by the spatial detail module can not only enhance the depth perception information, but also be used to generate multi-plane image features in subsequent steps.

[0065] like Figure 4 As shown, the depth enhancement module is used to fuse the coarse-grained depth features and the spatial detail features to generate fine-grained depth features, and decode the fine-grained depth features into a planar hierarchical mask. In this embodiment, the depth enhancement module includes an attention extraction block, a feature fusion block, and a mask decoding block.

[0066] The attention extraction block extracts the attention features from the coarse-grained depth features. Specifically, the attention extraction block includes: a convolution layer with a convolution kernel size of 3×3 and a step size of 1, a batch normalization layer, an upsampling layer, and a Sigmoid activation function layer. In this embodiment, the upsampling layer is used to enlarge the length and width of the coarse-grained depth features by 4 times to ensure that they are the same size as the spatial detail features. The Sigmoid activation function layer is used to output the final attention map.

[0067] The feature fusion block fuses the attention feature and the spatial detail feature to obtain the fine-grained depth feature. Specifically, the feature fusion block includes: 2 convolution layers with a convolution kernel size of 3×3 and a step size of 1, and 1 batch normalization layer. In this embodiment, the attention map is multiplied with the spatial detail feature and then sent to the feature fusion block for fusion and output of the fine-grained depth feature. The fine-grained depth feature is expressed as:

[0068]

[0069] Among them, d f represents the fine-grained deep features, F fusion Represents the depth enhancement module.

[0070] The mask decoding block decodes the fine-grained depth features into a planar hierarchical mask. Specifically, the mask decoding block includes: a convolution layer with a convolution kernel size of 3×3 and a step size of 1, a batch normalization layer, a PreRelu activation function layer, an upsampling layer, and a Softmax layer. In this embodiment, the upsampling layer is used to enlarge the feature length and width to the original video frame size. The Softmax layer is used to output the final planar hierarchical mask. In this embodiment, the planar hierarchical mask is represented as:

[0071]

[0072] Where i represents a natural number, N represents the number of plane layers, and M i Represents a planar layering mask.

[0073] The mask decoding block decodes the fine-grained depth features into planar hierarchical masks, and hierarchizes the planar image into multi-level images through the planar hierarchical masks.

[0074] Therefore, in this embodiment, the coarse-grained depth feature is used to represent the depth perception information of the video frame, and the fine-grained depth feature can perceive the detail information in the video frame more accurately by supplementing the spatial detail feature.

[0075] like Figure 5As shown, the multi-plane generation module is used to multiply the spatial detail features and the plane layered masks to obtain multi-plane image features, and obtain the multi-plane image by decoding the multi-plane image features. In this embodiment, the multi-plane generation module includes an upsampling unit, and the spatial detail features and the plane layered masks are fused to generate multi-plane image features, and the multi-plane image features are decoded after passing through the upsampling unit to obtain the multi-plane image. Specifically, the multi-plane generation module includes three upsampling units. The structure of each upsampling unit includes: 1 pixel shuffling layer, 2 convolution layers with a convolution kernel size of 3×3 and a step size of 1. It should be noted that since the resolution length and width of the spatial detail features are one-eighth of the original video frame, each upsampling unit reduces the number of channels of the features to one-fourth through the pixel shuffling layer, but the length and width of the feature resolution are magnified by two times. Therefore, the multi-plane image features are decoded after passing through the upsampling unit to obtain a multi-plane image, and the multi-plane image is expressed as:

[0076]

[0077] Among them, P i represents a multi-plane image, F RGBα Represents the multi-plane generation module.

[0078] The output module forms a stereoscopic video frame through multi-plane images and outputs the stereoscopic video frame. It should be noted that for the generated multi-plane images, it is also necessary to perform a plane transformation on the multi-plane images through a homography transformation formula to render a new perspective. In this embodiment, the homography transformation formula is expressed as:

[0079]

[0080] Among them, x s Indicates the horizontal coordinate of the source perspective pixel, y s represents the vertical coordinate of the pixel point of the source perspective, t represents the translation vector between perspectives, n represents the plane normal vector, which is [0,0,1] here, z n Indicates the plane depth value, x t Indicates the horizontal coordinate of the target viewing angle pixel, y t Represents the ordinate of the pixel point of the target viewing angle, T represents the transposition operation, K represents the camera intrinsic parameter, and R represents the rotation matrix between viewing angles.

[0081] Therefore, compared with the traditional method that usually requires a depth map to provide auxiliary information, the present embodiment provides a two-dimensional video stereoscopic conversion system. The present embodiment obtains depth perception by constructing a lightweight depth perception module, thereby saving time and labor costs; the spatial detail features extracted by constructing a spatial detail module can not only enhance the depth perception information, but also be used to generate multi-plane image features; the depth information is enhanced by utilizing detail features by constructing a depth enhancement module, thereby obtaining accurate depth perception information and local details; and a multi-level image representation is generated by constructing a multi-plane generation module, thereby realizing real-time two-dimensional video stereoscopic conversion.

[0082] The present embodiment also provides a method for converting a planar video into a stereoscopic form, comprising the following steps: acquiring and inputting a planar video; extracting coarse-grained depth features of video frames in the planar video; extracting spatial detail features of video frames in the planar video; fusing the coarse-grained depth features and the spatial detail features to generate fine-grained depth features, and decoding the fine-grained depth features into a planar layered mask; fusing the spatial detail features and the planar layered mask to obtain multi-plane image features, and obtaining a multi-plane image by decoding the multi-plane image features; constructing a stereoscopic video frame by using the multi-plane image, and outputting the stereoscopic video frame.

[0083] Therefore, the present embodiment provides a method for converting a two-dimensional video into a three-dimensional video, which utilizes multi-level images to construct a three-dimensional expression, and can quickly and accurately realize the three-dimensionalization of a two-dimensional video. It has high robustness, strong generalization, strong practicality and wide applicability.

[0084] This embodiment also provides a computer device. A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the psychological stress identification method when executing the computer program.

[0085] This embodiment also provides a computer-readable storage medium. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method for identifying psychological stress described in this embodiment is executed. The computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device; the program code contained in the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0086] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.

Claims

1. A system for converting a flat video into a three-dimensional video, characterized in that: include: Input module, obtains and inputs planar video; Depth perception module, extracts coarse-grained depth features of video frames in planar videos; The spatial detail module extracts the spatial detail features of video frames in the planar video; The depth enhancement module fuses the coarse-grained depth features and the spatial detail features to generate fine-grained depth features, and decodes the fine-grained depth features into planar hierarchical masks; The multi-plane generation module multiplies the spatial detail features and the plane layered mask to obtain the multi-plane image features, and obtains the multi-plane image by decoding the multi-plane image features; The output module forms a stereoscopic video frame through multi-plane images and outputs the stereoscopic video frame.

2. A planar video stereo conversion system according to claim 1, characterized in that: The depth perception module includes a feature extraction layer and a basic block unit; the feature extraction layer decomposes the video frame into a feature map, and the basic block unit extracts features from the feature map to obtain a coarse-grained depth feature; the coarse-grained depth feature is expressed as: Among them, d c represents the coarse-grained depth feature, I represents the video frame; F depth Represents the depth perception module.

3. A planar video stereo conversion system according to claim 2, characterized in that: The spatial detail module includes a spatial feature extraction block, which extracts spatial detail features in the video frame; the spatial detail features are expressed as: Among them, f d Indicates that F detail Represents the spatial detail module.

4. The system for converting a two-dimensional video into a three-dimensional video according to claim 3, characterized in that: The depth enhancement module includes an attention extraction block, a feature fusion block and a mask decoding block; the attention extraction block extracts attention features from coarse-grained depth features, and the feature fusion block fuses attention features and spatial detail features to obtain fine-grained depth features; the fine-grained depth features are expressed as: Among them, d f represents the fine-grained deep features, F fusion Represents the depth enhancement module.

5. A planar video stereo conversion system according to claim 4, characterized in that: The mask decoding block decodes the fine-grained depth features into a planar hierarchical mask; the planar hierarchical mask is represented as: Where i represents a natural number, N represents the number of plane layers, and M i Represents a planar layering mask.

6. The system for converting a two-dimensional video into a three-dimensional video according to claim 5, characterized in that: The multi-plane generation module includes an upsampling unit; the spatial detail features and the plane layered mask are fused to generate multi-plane image features, and the multi-plane image features are decoded by the upsampling unit to obtain a multi-plane image; the multi-plane image is represented as: Among them, P i represents a multi-plane image, F RGBα Represents the multi-plane generation module.

7. The system for converting a two-dimensional video into a three-dimensional video according to claim 6, characterized in that: The multi-plane image is transformed into a plane using a homography transformation formula; the homography transformation formula is expressed as: Among them, x s Indicates the horizontal coordinate of the source perspective pixel, y s represents the vertical coordinate of the pixel point of the source perspective, t represents the translation vector between perspectives, n represents the plane normal vector, z n Indicates the plane depth value, x t Indicates the horizontal coordinate of the target viewing angle pixel, y t Represents the ordinate of the pixel point of the target viewing angle, T represents the transposition operation, K represents the camera intrinsic parameter, and R represents the rotation matrix between viewing angles.

8. A method for converting a two-dimensional video into a three-dimensional video, comprising the following steps: Get and input the flat video; Extracting coarse-grained deep features of video frames in planar videos; Extract spatial detail features of video frames in planar video; Fusing coarse-grained depth features and spatial detail features to generate fine-grained depth features, and decoding the fine-grained depth features into planar hierarchical masks; The multi-plane image features are obtained by fusing the spatial detail features and the plane layered mask, and the multi-plane image is obtained by decoding the multi-plane image features; A stereoscopic video frame is formed by using multi-plane images, and the stereoscopic video frame is output.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to claim 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in claim 8 is performed.

Images