Free viewpoint map generation method, device, equipment and storage medium

Abstract

The present application provides a free viewpoint image generation method, device, equipment and storage medium, the method comprising: extracting the features of the multi-viewpoint image by combining the internal and external parameters of the multi-viewpoint image through a feature extraction network, and obtaining the final target viewpoint depth map by combining the unsupervised stereo matching network; extracting the features of the multi-viewpoint image by a convolutional neural network to obtain multiple depth coding images to be processed of the multi-viewpoint image; projecting each depth coding image to be processed by combining the final target viewpoint depth map by a DIBR method to obtain multiple target depth coding images; fusing each target depth coding image by a preset aggregation module to obtain a target viewpoint coding image, and decoding the target viewpoint coding image by a fully convolutional network to obtain a target free viewpoint image. The free viewpoint image generation method provided in the embodiment of the present application generates a virtual target free viewpoint image from a multi-viewpoint image through depth estimation, so that the target free viewpoint image has high accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing and computer vision technology, and in particular to a method, device, equipment and storage medium for generating a free viewpoint map. Background Art

[0002] Traditional two-dimensional image acquisition and display technology is difficult to meet the growing viewing needs of viewers. With the innovation of display technology and the improvement of computing power, methods for accurately and efficiently reproducing three-dimensional light field information have received a lot of attention. In order to reproduce the light field information of real scenes in a true three-dimensional light field display, it is necessary to build a camera array to capture the real scene and restore the three-dimensional light field information through dense viewpoints. In this process, due to the physical limitations of the camera, it is difficult to perform dense viewpoint acquisition, and only sparse viewpoint acquisition can be satisfied. In order to generate dense viewpoint content, researchers have proposed many virtual viewpoint generation methods that use sparse viewpoints as input to generate dense viewpoints.

[0003] Existing methods based on multi-plane images work well when the input viewpoints are arranged horizontally, but when the input viewpoint positions are rotated, such methods are difficult to generate accurate results. Although the existing methods based on stereo mesh scaffold can effectively handle the problem of input viewpoint positions containing rotations, this method requires a large number of scene images (more than 50) to be input in advance and is extremely time-consuming (on the order of hours).

[0004] That is to say, when the collected input viewpoint images are relatively sparse, the camera positions contain large rotations and the arrangement has no obvious regularity, it is difficult for the existing virtual viewpoint generation methods to generate accurate target free viewpoint maps. Summary of the invention

[0005] The present application provides a free viewpoint map generation method, device, equipment and storage medium, aiming to generate a target free viewpoint map with high accuracy.

[0006] In a first aspect, the present application provides a method for generating a free viewpoint graph, comprising:

[0007] The features of the multi-view image are extracted by combining the internal and external parameters of the multi-view image through a feature extraction network, and the final target view depth image is obtained by combining the unsupervised stereo matching network;

[0008] Performing feature extraction on the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image;

[0009] Projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images;

[0010] The target depth coding images are fused through a preset aggregation module to obtain a target viewpoint coding image, and the target viewpoint coding image is decoded through a full convolutional network to obtain a target free viewpoint image.

[0011] In one embodiment, the extracting features of the multi-view image by combining the internal and external parameters of the multi-view image through a feature extraction network, and obtaining the final target view depth map by combining an unsupervised stereo matching network, includes:

[0012] Determining a multi-view geometric relationship between the multi-view images according to internal and external parameters of the multi-view images;

[0013] By combining the multi-view geometric relationship with the feature extraction network, feature extraction is performed on the multi-view image to obtain scale feature images of different resolutions of a preset group of multiple viewpoints;

[0014] Uniformly sampling in the depth direction according to the depth distribution range of the multi-view image to generate multiple depth planes;

[0015] The preset group of scale feature maps of different resolutions of the multiple viewpoints are transformed to each of the depth planes to obtain each target depth plane, and the final target viewpoint depth map is obtained according to each of the target depth planes.

[0016] The preset group of scale feature maps of different resolutions of the multiple viewpoints includes a scale feature map of a quarter resolution,

[0017] The step of transforming the scale feature maps of the preset group of different resolutions of the multiple viewpoints to each of the depth planes to obtain each of the target depth planes, and obtaining the final target viewpoint depth map according to each of the target depth planes, includes:

[0018] Constructing a first homography transformation matrix in each depth plane, and transforming the scale feature maps of a quarter of the resolution of the multiple viewpoints to each depth plane through each of the first homography transformation matrices to obtain each first target depth plane;

[0019] Combining the first target depth planes according to the depth order to construct a first matching cost volume;

[0020] Matching the first matching cost volume through a stereo convolutional neural network to obtain a first probability value that each region in the cost volume space belongs to a three-dimensional object;

[0021] Normalizing the first probability value by a preset function, and weighted superimposing the normalized first probability value and its depth value to obtain the first target viewpoint depth map;

[0022] If the resolution of the first target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the first target viewpoint depth map is determined as the final target viewpoint depth map.

[0023] The preset group of scale feature maps of different resolutions of the multiple viewpoints includes a scale feature map with half resolution,

[0024] After the first probability value is normalized by a preset function and the normalized first probability value and its depth value are weightedly superimposed to obtain the first target viewpoint depth map, the method further includes:

[0025] If the resolution of the first target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the first target viewpoint depth map as a first initial value, and determining a first depth search range according to the first initial value and a first preset initial difference;

[0026] Constructing a second homography transformation matrix in each depth plane according to the first depth search range and the internal and external parameters of the multiple viewpoints;

[0027] Transforming the scale feature maps of the plurality of viewpoints at half the resolution to respective depth planes through the respective second homography transformation matrices to obtain respective second target depth planes;

[0028] Combining each of the second target depth planes according to the depth order to construct a second matching cost volume;

[0029] Matching the second matching cost volume by the stereo convolutional neural network to obtain a second probability value that each region in the cost volume space belongs to a three-dimensional object;

[0030] Normalizing the second probability value by using the preset function, performing weighted superposition on the normalized second probability value and its depth value to obtain a second target viewpoint depth map;

[0031] If the resolution of the second target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the second target viewpoint depth map is determined as the final target viewpoint depth map.

[0032] The preset group of scale feature maps of different resolutions of the multiple viewpoints includes a scale feature map of double resolution,

[0033] After normalizing the second probability value by the preset function and weighted superpositioning the normalized second probability value and its depth value to obtain a second target viewpoint depth map, the method further includes:

[0034] If the resolution of the second target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the second target viewpoint depth map as a second initial value, and determining a second depth search range according to the second initial value and a second preset initial difference value;

[0035] constructing a third homography transformation matrix in each depth plane according to the second depth search range and the internal and external parameters of the plurality of viewpoints;

[0036] By using each of the third homography transformation matrices, the scale feature maps of the multiple viewpoints with one times the resolution are transformed to each depth plane to obtain each third target depth plane;

[0037] Combining the third target depth planes according to the depth order to construct a third matching cost volume;

[0038] Matching the third matching cost volume by using the stereo convolutional neural network to obtain a third probability value that each region in the cost volume space belongs to a three-dimensional object;

[0039] Normalizing the third probability value by using the preset function, performing weighted superposition on the normalized third probability value and its depth value to obtain a third target viewpoint depth map;

[0040] If the resolution of the third target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the third target viewpoint depth map is determined as the final target viewpoint depth map.

[0041] The step of extracting features from the multi-view image by a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image includes:

[0042] Performing feature extraction on the multi-view image through a VGG convolutional neural network, and outputting feature images of the first three stages, wherein the sizes of the feature images of the first three stages are the original size, half size, and one quarter size of the multi-view image, respectively;

[0043] Perform nearest neighbor interpolation on the half-size and quarter-size feature maps in the first three stages, and upsample them to a resolution consistent with the multi-view image;

[0044] The processed half-size feature map and the processed quarter-size feature map are spliced ​​with the original-size feature map in the feature dimension to generate the depth coding maps to be processed.

[0045] The method of projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by the DIBR method to obtain a plurality of target depth coding images includes:

[0046] In combination with the final target viewpoint depth map and its corresponding virtual camera parameters, each of the to-be-processed depth coding maps is projected to the target viewpoint through the DIBR method to obtain each of the target depth coding maps.

[0047] In a second aspect, the present application provides a free viewpoint map generating device, comprising:

[0048] A first extraction module is used to extract features of the multi-view image by combining internal and external parameters of the multi-view image through a feature extraction network, and to obtain a final target view depth map by combining an unsupervised stereo matching network;

[0049] A second extraction module is used to extract features from the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image;

[0050] A projection module, configured to project each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images;

[0051] The fusion decoding module is used to fuse the target depth coding images through a preset aggregation module to obtain a target viewpoint coding image, and to decode the target viewpoint coding image through a fully convolutional network to obtain a target free viewpoint image.

[0052] In a third aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the free viewpoint map generation method described in the first aspect when executing the program.

[0053] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program, and when the computer program is executed by the processor, the steps of the free viewpoint map generation method described in the first aspect are implemented.

[0054] In a fifth aspect, the present application also provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the steps of the free viewpoint map generation method described in the first aspect are implemented.

[0055] The free viewpoint map generation method, device, equipment and storage medium provided by the present application perform depth estimation on the multi-viewpoint map through the feature extraction network and the internal and external parameters of the multi-viewpoint map during the free viewpoint map generation process, and combine the depth map obtained by the depth estimation with the convolutional neural network, the DIBR method, the preset aggregation module and the full convolutional network. The multi-viewpoint map with sparse viewpoints is combined with multiple neural networks through depth estimation to generate a virtual target free viewpoint map with dense viewpoints. That is, the present application realizes both the depth estimation and viewpoint generation stages by neural networks, and then generates a target free viewpoint map with high accuracy through joint optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 It is one of the flowcharts of the free viewpoint graph generation method provided by this application;

[0058] Figure 2 It is a schematic diagram of the unsupervised learning network structure of depth estimation of the free viewpoint graph generation method provided by this application;

[0059] Figure 3 This is the second flow chart of the free viewpoint graph generation method provided by this application;

[0060] Figure 4 This is the third flow chart of the free viewpoint graph generation method provided by this application;

[0061] Figure 5 This is the fourth flowchart of the free viewpoint graph generation method provided by this application;

[0062] Figure 6 This is the fifth flow chart of the free viewpoint graph generation method provided by this application;

[0063] Figure 7 It is a structural schematic diagram of a free viewpoint map generating device provided by the present application;

[0064] Figure 8 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] Combine the following Figures 1 to 8 The present invention describes a method, device, equipment and storage medium for generating a free viewpoint map.

[0067] Reference Figures 1 to 8 , Figure 1 It is one of the flowcharts of the free viewpoint graph generation method provided by this application; Figure 2 It is a schematic diagram of the unsupervised learning network structure of depth estimation of the free viewpoint graph generation method provided by this application; Figure 3 This is the second flow chart of the free viewpoint graph generation method provided by this application; Figure 4 This is the third flow chart of the free viewpoint graph generation method provided by this application; Figure 5 This is the fourth flowchart of the free viewpoint graph generation method provided by this application; Figure 6 This is the fifth flow chart of the free viewpoint graph generation method provided by this application; Figure 7 It is a structural schematic diagram of a free viewpoint map generating device provided by the present application; Figure 8 It is a structural schematic diagram of the electronic device provided by this application.

[0068] The embodiments of the present application provide embodiments of a method for generating a free viewpoint map. It should be noted that, although a logical order is shown in the flowchart, under certain data, the steps shown or described may be completed in an order different from that shown here.

[0069] Glossary:

[0070] Application scenario: In a sparse viewpoint acquisition scenario (the default input is 5 viewpoints), generate virtual viewpoint images of positions adjacent to the input camera position.

[0071] Target viewpoint: The virtual viewpoint to be generated, determined by the three parameters K, R, and T.

[0072] Depth Estimation: Take sparse viewpoints as input and estimate the depth map of the target viewpoint.

[0073] DIBR: full name Depth-image-based rendering. For example, the color image of viewpoint 1 is projected to viewpoint 2 according to the spatial geometric relationship through the depth map of viewpoint 1 and the pose relationship between viewpoints 1 and 2, thereby generating the color image of viewpoint 2.

[0074] The embodiments of the present application take an electronic device as an example of an execution subject. The embodiments of the present application take a free viewpoint generation system as one of the forms of the electronic device, but do not limit the electronic device.

[0075] Specifically, refer to Figure 1 , Figure 1 This is one of the flow charts of the free viewpoint map generation method provided by the present application. The free viewpoint map generation method provided by the embodiment of the present application includes:

[0076] Step S10, extracting features of the multi-view image by combining the internal and external parameters of the multi-view image through a feature extraction network, and obtaining a final target view depth map by combining an unsupervised stereo matching network.

[0077] It should be noted that when a free viewpoint map with dense viewpoints needs to be generated, a multi-viewpoint map needs to be collected and input into the free viewpoint generation system. At this time, the multi-viewpoint map is a multi-viewpoint map composed of sparse viewpoints. Furthermore, a multi-viewpoint map does not refer to a single image, but multiple different images of an object from multiple perspectives. At the same time, a multi-viewpoint map is also a 2D (two dimensional) plane map.

[0078] The free viewpoint generation system reads the input multi-view image and the internal and external parameters of the multi-view image, where the external parameters include but are not limited to the K parameter and the T parameter, and the internal parameters include but are not limited to the R parameter. The external parameters ([R|T]) describe the transformation relationship between the world coordinate system and the camera coordinate system. The R parameter represents the rotation parameter, and the T parameter represents the translation parameter. The internal parameters (K) describe the transformation relationship between the camera coordinate system, the image coordinate system, and the pixel coordinate system. The K parameters include but are not limited to the principal point coordinates, focal length, unit pixel width, and unit pixel height. Next, the free viewpoint generation system extracts features from the multi-view image through a feature extraction network combined with the internal and external parameters of the multi-view image, and obtains the final target viewpoint depth map through an unsupervised stereo matching network, wherein the feature extraction network is a 2D multi-scale feature extraction network, that is, it can be understood that the free viewpoint generation system extracts features from the 2D multi-view image through a 2D multi-scale feature extraction network combined with the K parameters, T parameters and R parameters of the multi-view image, and obtains the final target viewpoint depth map, as described in steps S101 to S104.

[0079] Further, the description of step S101 to step S104 is as follows:

[0080] Step S101, determining a multi-view geometric relationship between the multi-view images according to internal and external parameters of the multi-view images;

[0081] Step S102, extracting features from the multi-view graph by combining the multi-view geometric relationship with the feature extraction network to obtain scale feature graphs of different resolutions of a preset group of multiple viewpoints;

[0082] Step S103, uniformly sampling in the depth direction according to the depth distribution range of the multi-view image to generate multiple depth planes;

[0083] Step S104 , transforming the preset group of scale feature maps of different resolutions of the multiple viewpoints to each of the depth planes to obtain each of the target depth planes, and obtaining the final target viewpoint depth map according to each of the target depth planes.

[0084] Specifically, the free viewpoint generation system first determines the multi-view geometric relationship between the multi-view images according to the internal and external parameters of the multi-view images. Then, the free viewpoint generation system extracts features from the input multi-view images by combining the multi-view geometric relationship between the multi-view images through a 2D multi-scale feature extraction network, and outputs scale feature images of different resolutions of a preset group of multiple viewpoints, where the minimum resolution can be 1 / (2 n-1 ), n is a preset group, and the preset group is determined by the number of images in the input multi-view image. The resolution of the scale feature map is relative to the resolution of the original image of multiple viewpoints. For example, if the resolution is 1 / 4 times, it can be understood that the resolution of the scale feature map is 1 / 4 of the resolution of the original image of multiple viewpoints. Further, in this embodiment, if the number of images in the multi-view image is 5, the preset groups are 5, and the 5 groups of different resolutions of the multiple viewpoints are 1 / 32 times, 1 / 16 times, 1 / 4 times, 1 / 2 times and 1 times, and 1 times means that the resolution of the scale feature map is the resolution of the original image of multiple viewpoints; for example, if the number of images in the multi-view image is 3, the preset groups are 3, and the 3 groups of different resolutions of the multiple viewpoints are 1 / 4 times, 1 / 2 times and 1 times,

[0085] Furthermore, the free viewpoint generation system determines the minimum depth value D of the multi-view image according to the acquisition scene of the multi-view image. min and the maximum depth value D max , according to the minimum depth value D min and the maximum depth value D max Determine the depth distribution range of the multi-view image [D min , D max Next, the free viewpoint generation system generates a viewpoint based on the depth distribution range [D min , D max ] uniformly samples in the depth direction to generate a series of depth planes. Then, the free viewpoint generation system transforms the scale feature maps of different resolutions of the preset groups of multiple viewpoints into each depth plane, obtains each transformed target depth plane, splices each target depth plane to obtain the final target viewpoint depth map, and combines the Figure 2 Conduct analysis, Figure 2 It is a schematic diagram of the unsupervised learning network structure for depth estimation of the free viewpoint graph generation method provided in this application.

[0086] This embodiment takes the number of images of the multi-view image as 3 as an example, specifically: the free viewpoint generation system reads the input image, i.e., the multi-view image I1, i.e., the multi-view image I2 and i.e., the multi-view image I3, the width of the multi-view image I1, i.e., the multi-view image I2 and i.e., the multi-view image I3 is W, and the height is H, and performs feature extraction through the 2D multi-scale feature extraction network to obtain the multi-view image I1, i.e., the scale feature map of 1 / 4 times the scale of the multi-view image I2 and i.e., the scale feature map of the width is H / 4 and the height is W / 4, and the scale feature map of the multi-view image I1, i.e., the scale feature map of 1 / 2 times the scale of the multi-view image I2 and i.e., the scale feature map of the width is H / 2 and the height is W / 2, and the scale feature map of the multi-view image I1, i.e., the scale feature map of 1 times the scale of the multi-view image I2 and i.e., the scale feature map of the width is H and the height is W.

[0087] Further, the free viewpoint generation system transforms the scale feature maps of different resolutions of the preset group of multiple viewpoints into each depth plane in the order of minimum resolution to maximum resolution. Therefore, the free viewpoint generation system transforms the feature map with the lowest resolution into each depth plane. This embodiment can be understood as the free viewpoint generation system transforming the scale feature map with a width of H / 4 and a height of W / 4 into each depth plane, and then constructing the first stage 3D matching cost volume according to the transformed target depth plane, and then matching the first stage 3D matching cost volume through 3D CNN (three Dimensional Convolutional Neural Networks, 3D convolutional neural network) to generate the first stage depth map. Finally, the free viewpoint generation system determines whether the first stage depth map meets the requirements. If it is determined that the first stage depth map meets the requirements, the free viewpoint generation system determines the first stage depth map as the final target viewpoint depth map, as described in steps S1041 to S1045.

[0088] Furthermore, if it is determined that the first-stage depth map does not meet the requirements, the free viewpoint generation system transforms the feature map of the intermediate resolution to each depth plane. This embodiment can be understood as the free viewpoint generation system transforming the scale feature map with a width of H / 2 and a height of W / 2 to each depth plane, and then constructing the second-stage 3D matching cost volume based on the transformed target depth plane, and then matching the second-stage 3D matching cost volume through 3D CNN to generate a second-stage depth map. Finally, the free viewpoint generation system determines whether the second-stage depth map meets the requirements. If it is determined that the second-stage depth map meets the requirements, the free viewpoint generation system determines the second-stage depth map as the final target viewpoint depth map, as described in steps S1046 to S10412.

[0089] Furthermore, if it is determined that the second-stage depth map does not meet the requirements, the free viewpoint generation system transforms the highest resolution feature map to each depth plane. This embodiment can be understood as the free viewpoint generation system transforming the scale feature map with a width of H and a height of W to each depth plane, and then constructing the third-stage 3D matching cost volume according to the transformed target depth plane, and then matching the third-stage 3D matching cost volume through 3D CNN to generate the third-stage depth map. Finally, the free viewpoint generation system determines whether the third-stage depth map meets the requirements. If it is determined that the third-stage depth map meets the requirements, the free viewpoint generation system determines the third-stage depth map as the final target viewpoint depth map, as described in steps S10413 to S10419.

[0090] Furthermore, the embodiment of the present application minimizes the error function and optimizes the network parameters through unsupervised learning. Therefore, the embodiment of the present application reduces the dependence on deep label data and enhances the generalization of the model.

[0091] The embodiment of the present application combines depth estimation with a neural network to ensure the accuracy of depth estimation.

[0092] Step S20: extracting features from the multi-view image through a convolutional neural network to obtain a plurality of depth coding images to be processed of the multi-view image.

[0093] It should be noted that the convolutional neural network in this embodiment can be a VGG-16 convolutional neural network or a VGG-19 convolutional neural network.

[0094] Therefore, the free viewpoint generation system extracts features from the input sparse viewpoint multi-view image through the VGG-16 or VGG-19 convolutional neural network, and outputs multiple deep coding images E of the multi-view image to be processed. i , as described in steps S201 to S203.

[0095] Step S30, projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map through a DIBR method to obtain a plurality of target depth coding images.

[0096] The free viewpoint generation system performs DIBR transformation on each depth coding image to be processed by combining the final target viewpoint depth map through the DIBR method, projects the sparse viewpoint to the target viewpoint, and obtains each target depth coding image, as shown in step a, wherein the DIBR method is a geometric calculation method.

[0097] Furthermore, in step a, in combination with the final target viewpoint depth map and its corresponding virtual camera parameters, each of the to-be-processed depth coding images is projected to the target viewpoint through the DIBR method to obtain each of the target depth coding images.

[0098] Specifically, the free viewpoint generation system determines the final target viewpoint depth map and its corresponding virtual camera parameters, and the virtual camera parameters include but are not limited to K tgt Parameters, R tgt Parameters and T tgt Then, the free viewpoint generation system is based on the final target viewpoint depth map and its K tgt Parameters, R tgt Parameters and T tgt Parameters, and combined with the DIBR method, DIBR transformation is performed on each depth coding image to be processed, and the sparse viewpoint is projected to the target viewpoint to obtain each target depth coding image E i→gtg , the specific transformation process is described in Formula 1 and Formula 2.

[0099] Formula 1:

[0100] Formula 2: E i→tgt (p) = E i (p' i )

[0101] Among them, P is the coordinate of any pixel in the target viewpoint image coordinate system, D is the depth value of the current pixel, KRT is the position parameter of the multi-view image, and p' i is the coordinate value of the pixel p in the target viewpoint coordinate system projected to the multi-view image coordinate system, E i is the depth coding image to be processed.

[0102] The embodiment of the present application combines the generation of the target depth coding map with the neural network, thereby ensuring the accuracy of the target depth coding map.

[0103] Step S40, fusing the target depth coding images through a preset aggregation module to obtain a target viewpoint coding image, and decoding the target viewpoint coding image through a fully convolutional network to obtain a target free viewpoint image.

[0104] It should be noted that the preset aggregation module in this embodiment is a self-attention-ray aggregation module. Therefore, the free viewpoint generation system fuses the target depth coding map located at the same pixel coordinate position into a feature vector through the self-attention-ray aggregation module to obtain a target viewpoint coding map. Next, the free viewpoint generation system decodes the target viewpoint coding map through multiple cascaded U-Net full convolutional networks to generate a target free viewpoint map with dense free viewpoints. The target free viewpoint map is a 2D color map. Finally, the free viewpoint generation system also needs to calculate the generated target free viewpoint map with the real target free viewpoint map, and use the calculated loss Loss to train the neural network and update the network parameters.

[0105] The present embodiment provides a method for generating a free viewpoint map. During the free viewpoint map generation process, the depth of the multi-viewpoint map is first estimated through a feature extraction network and internal and external parameters of the multi-viewpoint map. The depth map obtained by the depth estimation is then combined with a convolutional neural network, a DIBR method, a preset aggregation module and a full convolutional network. The multi-viewpoint map with sparse viewpoints is combined with multiple neural networks through depth estimation to generate a virtual target free viewpoint map with dense viewpoints. That is, the present application realizes both the depth estimation and viewpoint generation stages by neural networks, and then generates a target free viewpoint map with high accuracy through joint optimization.

[0106] Reference Figure 3 , Figure 3 This is the second flow chart of the free viewpoint map generation method provided by the present application, step S104 includes:

[0107] Step S1041, constructing a first homography transformation matrix in each depth plane, and transforming the scale feature maps of the quarter-resolution of the multiple viewpoints to each depth plane through each of the first homography transformation matrices to obtain each first target depth plane;

[0108] Step S1042, combining the first target depth planes according to the depth order to construct a first matching cost volume;

[0109] Step S1043, matching the first matching cost volume through a stereo convolutional neural network to obtain a first probability value that each region in the cost volume space belongs to a three-dimensional object;

[0110] Step S1044, normalizing the first probability value by a preset function, performing weighted superposition on the normalized first probability value and its depth value to obtain the first target viewpoint depth map;

[0111] Step S1045: If the resolution of the first target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the first target viewpoint depth map is determined as the final target viewpoint depth map.

[0112] Specifically, the preset group of scale feature maps of different resolutions of multiple viewpoints in this embodiment includes a scale feature map with a quarter resolution, that is, a scale feature map with a width of H / 4 and a height of W / 4. Therefore, the free viewpoint generation system constructs the first homography transformation matrix of each depth plane in each depth plane, and transforms the scale feature map with a width of H / 4 and a height of W / 4 to the respective depth planes through the first homography transformation matrix of each depth plane to obtain the transformed first target depth planes. Then, the free viewpoint generation system combines the first target depth planes according to the depth order to construct a first 3D matching cost volume, wherein the depth order can be from high to low or from low to high. Next, the free viewpoint generation system matches the first 3D matching cost volume through a 3D stereo convolutional neural network to obtain the first probability value that each area in the cost volume space belongs to a three-dimensional object. Next, the free viewpoint generation system normalizes the first probability value through a preset Softmax function to obtain a normalized first probability value, and then weightedly superimposes the normalized first probability value and its corresponding depth value to obtain a first target viewpoint depth map. Finally, the free viewpoint generation system determines whether the resolution of the first target viewpoint depth map is equal to the resolution of the multi-view map. If it is determined that the resolution of the first target viewpoint depth map is equal to the resolution of the multi-view map, the free viewpoint generation system determines the first target viewpoint depth map as the final target viewpoint depth map.

[0113] The embodiment of the present application combines depth estimation with a neural network, thereby ensuring the accuracy of the depth estimation and further ensuring the accuracy of the target free viewpoint map.

[0114] Reference Figure 4 , Figure 4 This is the third flow chart of the free viewpoint map generation method provided by the present application. After step S1044, it also includes:

[0115] Step S1046: if the resolution of the first target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the first target viewpoint depth map as a first initial value, and determining a first depth search range according to the first initial value and a first preset initial difference;

[0116] Step S1047, constructing a second homography transformation matrix in each depth plane according to the first depth search range and the internal and external parameters of the multiple viewpoints;

[0117] Step S1048, transforming the scale feature maps of the plurality of viewpoints with half the resolution to respective depth planes through the respective second homography transformation matrices to obtain respective second target depth planes;

[0118] Step S1049, combining the second target depth planes according to the depth order to construct a second matching cost volume;

[0119] Step S10410, matching the second matching cost volume by the stereo convolutional neural network to obtain a second probability value that each region in the cost volume space belongs to a three-dimensional object;

[0120] Step S10411, normalizing the second probability value by the preset function, performing weighted superposition on the normalized second probability value and its depth value to obtain a second target viewpoint depth map;

[0121] Step S10412: If the resolution of the second target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the second target viewpoint depth map is determined as the final target viewpoint depth map.

[0122] Specifically, the preset group of scale feature maps of different resolutions of multiple viewpoints in this embodiment includes a scale feature map with half resolution, that is, a scale feature map with a width of H / 2 and a height of W / 2. Therefore, if it is determined that the resolution of the first target viewpoint depth map is less than the resolution of the multi-viewpoint map, the free viewpoint generation system uses the first target viewpoint depth map as the first initial value D1, and determines the first preset initial difference Δ1 of the first initial value D1, and obtains the minimum search value according to the difference between the first initial value D1 and the first preset initial difference Δ1, that is, the minimum search value is D1-Δ1, and obtains the maximum search value according to the sum of the first initial value D1 and the first preset initial difference Δ1, that is, the maximum search value is D1+Δ1, and then determines the first depth search range according to the minimum search value and the maximum search value, that is, the first depth search range is [D1-Δ1, D1+Δ1]. Next, the free viewpoint generation system constructs a second homography transformation matrix in each depth plane according to the first depth search range [D1-Δ1, D1+Δ1], as well as the K parameters, R parameters and T parameters of multiple viewpoints. Next, the free viewpoint generation system transforms the scale feature map with a width of H / 2 and a height of W / 2 to the respective depth planes through the second homography transformation matrix of each depth plane to obtain each transformed second target depth plane. Next, the free viewpoint generation system combines each second target depth plane according to the depth order to construct a second 3D matching cost volume, wherein the depth order can be from high to low or from low to high. Next, the free viewpoint generation system matches the second 3D matching cost volume through a 3D stereo convolutional neural network to obtain a second probability value that each area in the cost volume space belongs to a three-dimensional object. Next, the free viewpoint generation system normalizes the second probability value through a preset Softmax function to obtain a normalized second probability value, and then weightedly superimposes the normalized second probability value and its corresponding depth value to obtain a second target viewpoint depth map. Finally, the free viewpoint generation system determines whether the resolution of the second target viewpoint depth map is equal to the resolution of the multi-view map. If it is determined that the resolution of the second target viewpoint depth map is equal to the resolution of the multi-view map, the free viewpoint generation system determines the second target viewpoint depth map as the final target viewpoint depth map.

[0123] The embodiment of the present application combines depth estimation with a neural network, thereby ensuring the accuracy of the depth estimation and further ensuring the accuracy of the target free viewpoint map.

[0124] Reference Figure 5 , Figure 5 This is the fourth flowchart of the free viewpoint map generation method provided by the present application. After step S10411, it also includes:

[0125] Step S10413: if the resolution of the second target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the second target viewpoint depth map as a second initial value, and determining a second depth search range according to the second initial value and a second preset initial difference value;

[0126] Step S10414, constructing a third homography transformation matrix in each depth plane according to the second depth search range and the internal and external parameters of the multiple viewpoints;

[0127] Step S10415, transforming the scale feature maps of the multiple viewpoints with one times the resolution to each depth plane through each of the third homography transformation matrices to obtain each third target depth plane;

[0128] Step S10416, combining the third target depth planes according to the depth order to construct a third matching cost volume;

[0129] Step S10417, matching the third matching cost volume through the stereo convolutional neural network to obtain a third probability value that each region in the cost volume space belongs to a three-dimensional object;

[0130] Step S10418, normalizing the third probability value by the preset function, performing weighted superposition on the normalized third probability value and its depth value to obtain a third target viewpoint depth map;

[0131] Step S10419: If the resolution of the third target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the third target viewpoint depth map is determined as the final target viewpoint depth map.

[0132] Specifically, the preset group of scale feature maps of different resolutions of multiple viewpoints in this embodiment includes a scale feature map of one resolution, that is, a scale feature map with a width of H and a height of W. Therefore, if it is determined that the resolution of the second target viewpoint depth map is less than the resolution of the multi-viewpoint map, the free viewpoint generation system uses the second target viewpoint depth map as the second initial value D2, and determines the second preset initial difference Δ2 of the second initial value D2, obtains the minimum search value according to the difference between the second initial value D2 and the second preset initial difference Δ2, that is, the minimum search value is D2-Δ2, obtains the maximum search value according to the sum of the second initial value D2 and the second preset initial difference Δ2, that is, the maximum search value is D2+Δ2, and then determines the second depth search range according to the minimum search value and the maximum search value, that is, the second depth search range is [D2-Δ2, D2+Δ2]. Then, the free viewpoint generation system constructs a third homography transformation matrix in each depth plane according to the second depth search range [D2-Δ2, D2+Δ2], and the K parameters, R parameters and T parameters of multiple viewpoints. Next, the free viewpoint generation system transforms the scale feature map with a width of H and a height of W into the respective depth planes through the third homography transformation matrix of each depth plane, and obtains each transformed third target depth plane. Next, the free viewpoint generation system combines each third target depth plane according to the depth order to construct a third 3D matching cost volume, wherein the depth order can be from high to low or from low to high. Next, the free viewpoint generation system matches the third 3D matching cost volume through a 3D stereo convolutional neural network to obtain a third probability value that each area in the cost volume space belongs to a three-dimensional object. Next, the free viewpoint generation system normalizes the third probability value through a preset Softmax function to obtain a normalized third probability value, and then weightedly superimposes the normalized third probability value and its corresponding depth value to obtain a third target viewpoint depth map. Finally, the free viewpoint generation system determines whether the resolution of the third target viewpoint depth map is equal to the resolution of the multi-viewpoint map. If it is determined that the resolution of the third target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the free viewpoint generation system determines the third target viewpoint depth map as the final target viewpoint depth map.

[0133] The embodiment of the present application combines depth estimation with a neural network, thereby ensuring the accuracy of the depth estimation and further ensuring the accuracy of the target free viewpoint map.

[0134] Reference Figure 6 , Figure 6 This is the fifth flow chart of the free viewpoint map generation method provided by the present application, step S20 includes:

[0135] Step S201, extracting features from the multi-view image through a VGG convolutional neural network, and outputting feature images of the first three stages, wherein the sizes of the feature images of the first three stages are the original size, half size, and one quarter size of the multi-view image, respectively;

[0136] Step S202, performing nearest neighbor interpolation on the half-size and quarter-size feature maps in the feature maps of the first three stages, and upsampling them to a resolution consistent with the multi-view image;

[0137] Step S203, concatenating the processed half-size feature map and the processed quarter-size feature map with the original-size feature map in the feature dimension to generate the depth coding maps to be processed.

[0138] Specifically, the free viewpoint generation system extracts features from the multi-view image through a VGG-16 or VGG-19 convolutional neural network, and outputs feature maps of the first three stages, wherein the sizes of the feature maps of the first three stages are respectively equal to the original size of the multi-view image, half the size of the multi-view image, and one-quarter the size of the multi-view image. Then, the free viewpoint generation system performs nearest neighbor interpolation on the half-size feature map of the multi-view image in the feature maps of the first three stages, upsamples the resolution of the half-size feature map of the multi-view image to the same resolution as the multi-view image, and performs nearest neighbor interpolation on the quarter-size feature map of the multi-view image, upsamples the resolution of the quarter-size feature map of the multi-view image to the same resolution as the multi-view image. Finally, the free viewpoint generation system splices the processed half-size feature map and the processed quarter-size feature map with the original-size feature map in the feature dimension to generate each depth coding map to be processed.

[0139] The embodiment of the present application combines the generation process of the depth coding map to be processed with the neural network, thereby ensuring the accuracy of the depth coding map to be processed and further ensuring the accuracy of the target free viewpoint map.

[0140] Furthermore, the free viewpoint map generating device provided by the present application is described below, and the free viewpoint map generating device described below and the free viewpoint map generating method described above can refer to each other.

[0141] like Figure 7 As shown, Figure 7 : is a schematic diagram of the structure of the free viewpoint image generation device provided by the present application, and the free viewpoint image generation device comprises:

[0142] A first extraction module 701 is used to extract features of the multi-view image by combining internal and external parameters of the multi-view image through a feature extraction network, and to obtain a final target view depth map by combining an unsupervised stereo matching network;

[0143] A second extraction module 702 is used to extract features from the multi-view image through a convolutional neural network to obtain a plurality of depth coding images to be processed of the multi-view image;

[0144] A projection module 703 is used to project each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by using a DIBR method to obtain a plurality of target depth coding images;

[0145] The fusion decoding module 704 is used to fuse the target depth coding images through a preset aggregation module to obtain a target viewpoint coding image, and to decode the target viewpoint coding image through a full convolutional network to obtain a target free viewpoint image.

[0146] Furthermore, the first extraction module 701 is also used for:

[0147] Determining a multi-view geometric relationship between the multi-view images according to internal and external parameters of the multi-view images;

[0148] By combining the multi-view geometric relationship with the feature extraction network, feature extraction is performed on the multi-view image to obtain scale feature images of different resolutions of a preset group of multiple viewpoints;

[0149] Uniformly sampling in the depth direction according to the depth distribution range of the multi-view image to generate multiple depth planes;

[0150] The preset group of scale feature maps of different resolutions of the multiple viewpoints are transformed to each of the depth planes to obtain each target depth plane, and the final target viewpoint depth map is obtained according to each of the target depth planes.

[0151] Furthermore, the first extraction module 701 is also used for:

[0152] Constructing a first homography transformation matrix in each depth plane, and transforming the scale feature maps of a quarter of the resolution of the multiple viewpoints to each depth plane through each of the first homography transformation matrices to obtain each first target depth plane;

[0153] Combining the first target depth planes according to the depth order to construct a first matching cost volume;

[0154] Matching the first matching cost volume through a stereo convolutional neural network to obtain a first probability value that each region in the cost volume space belongs to a three-dimensional object;

[0155] Normalizing the first probability value by a preset function, and weighted superimposing the normalized first probability value and its depth value to obtain the first target viewpoint depth map;

[0156] If the resolution of the first target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the first target viewpoint depth map is determined as the final target viewpoint depth map.

[0157] Furthermore, the first extraction module 701 is also used for:

[0158] If the resolution of the first target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the first target viewpoint depth map as a first initial value, and determining a first depth search range according to the first initial value and a first preset initial difference;

[0159] Constructing a second homography transformation matrix in each depth plane according to the first depth search range and the internal and external parameters of the multiple viewpoints;

[0160] Transforming the scale feature maps of the plurality of viewpoints at half the resolution to respective depth planes through the respective second homography transformation matrices to obtain respective second target depth planes;

[0161] Combining each of the second target depth planes according to the depth order to construct a second matching cost volume;

[0162] Matching the second matching cost volume by the stereo convolutional neural network to obtain a second probability value that each region in the cost volume space belongs to a three-dimensional object;

[0163] Normalizing the second probability value by using the preset function, performing weighted superposition on the normalized second probability value and its depth value to obtain a second target viewpoint depth map;

[0164] If the resolution of the second target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the second target viewpoint depth map is determined as the final target viewpoint depth map.

[0165] Furthermore, the first extraction module 701 is also used for:

[0166] If the resolution of the second target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the second target viewpoint depth map as a second initial value, and determining a second depth search range according to the second initial value and a second preset initial difference value;

[0167] constructing a third homography transformation matrix in each depth plane according to the second depth search range and the internal and external parameters of the plurality of viewpoints;

[0168] By using each of the third homography transformation matrices, the scale feature maps of the multiple viewpoints with one times the resolution are transformed to each depth plane to obtain each third target depth plane;

[0169] Combining the third target depth planes according to the depth order to construct a third matching cost volume;

[0170] Matching the third matching cost volume by using the stereo convolutional neural network to obtain a third probability value that each region in the cost volume space belongs to a three-dimensional object;

[0171] Normalizing the third probability value by using the preset function, performing weighted superposition on the normalized third probability value and its depth value to obtain a third target viewpoint depth map;

[0172] If the resolution of the third target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the third target viewpoint depth map is determined as the final target viewpoint depth map.

[0173] Furthermore, the second extraction module 702 is further used for:

[0174] Performing feature extraction on the multi-view image through a VGG convolutional neural network, and outputting feature images of the first three stages, wherein the sizes of the feature images of the first three stages are the original size, half size, and one quarter size of the multi-view image, respectively;

[0175] Perform nearest neighbor interpolation on the half-size and quarter-size feature maps in the first three stages, and upsample them to a resolution consistent with the multi-view image;

[0176] The processed half-size feature map and the processed quarter-size feature map are spliced ​​with the original-size feature map in the feature dimension to generate the depth coding maps to be processed.

[0177] Furthermore, the projection module 703 is also used for:

[0178] In combination with the final target viewpoint depth map and its corresponding virtual camera parameters, each of the to-be-processed depth coding maps is projected to the target viewpoint through the DIBR method to obtain each of the target depth coding maps.

[0179] The specific embodiments of the free viewpoint map generation device provided in the present application are basically the same as the embodiments of the free viewpoint map generation method described above, and will not be described in detail here.

[0180] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the free viewpoint map generation method, which includes:

[0181] The features of the multi-view image are extracted by combining the internal and external parameters of the multi-view image through a feature extraction network, and the final target view depth image is obtained by combining the unsupervised stereo matching network;

[0182] Performing feature extraction on the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image;

[0183] Projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images;

[0184] The target depth coding images are fused through a preset aggregation module to obtain a target viewpoint coding image, and the target viewpoint coding image is decoded through a full convolutional network to obtain a target free viewpoint image.

[0185] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0186] On the other hand, the present application also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the free viewpoint map generation method provided by the above methods, the method comprising:

[0187] The features of the multi-view image are extracted by combining the internal and external parameters of the multi-view image through a feature extraction network, and the final target view depth image is obtained by combining the unsupervised stereo matching network;

[0188] Performing feature extraction on the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image;

[0189] Projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images;

[0190] The target depth coding images are fused through a preset aggregation module to obtain a target viewpoint coding image, and the target viewpoint coding image is decoded through a full convolutional network to obtain a target free viewpoint image.

[0191] In another aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the free viewpoint map generation method provided above is implemented, and the method includes:

[0192] The features of the multi-view image are extracted by combining the internal and external parameters of the multi-view image through a feature extraction network, and the final target view depth image is obtained by combining the unsupervised stereo matching network;

[0193] Performing feature extraction on the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image;

[0194] Projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images;

[0195] The target depth coding images are fused through a preset aggregation module to obtain a target viewpoint coding image, and the target viewpoint coding image is decoded through a full convolutional network to obtain a target free viewpoint image.

[0196] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0197] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a free viewpoint graph, characterized in that: include: The features of the multi-view image are extracted by combining the internal and external parameters of the multi-view image through a feature extraction network, and the final target view depth image is obtained by combining the unsupervised stereo matching network; Performing feature extraction on the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image; Projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images; The target depth coding images are fused through a preset aggregation module to obtain a target viewpoint coding image, and the target viewpoint coding image is decoded through a full convolutional network to obtain a target free viewpoint image.

2. The free viewpoint graph generation method according to claim 1, characterized in that: The method combines the internal and external parameters of the multi-view image through the feature extraction network to extract the features of the multi-view image, and combines the unsupervised stereo matching network to obtain the final target view depth map, including: Determining a multi-view geometric relationship between the multi-view images according to internal and external parameters of the multi-view images; By combining the multi-view geometric relationship with the feature extraction network, feature extraction is performed on the multi-view image to obtain scale feature images of different resolutions of a preset group of multiple viewpoints; Uniformly sampling in the depth direction according to the depth distribution range of the multi-view image to generate multiple depth planes; The preset group of scale feature maps of different resolutions of the multiple viewpoints are transformed to each of the depth planes to obtain each target depth plane, and the final target viewpoint depth map is obtained according to each of the target depth planes.

3. The free viewpoint graph generation method according to claim 2, characterized in that: The preset group of scale feature maps of different resolutions of the multiple viewpoints includes a scale feature map of a quarter resolution, The step of transforming the scale feature maps of the preset group of different resolutions of the multiple viewpoints to each of the depth planes to obtain each of the target depth planes, and obtaining the final target viewpoint depth map according to each of the target depth planes, includes: Constructing a first homography transformation matrix in each depth plane, and transforming the scale feature maps of a quarter of the resolution of the multiple viewpoints to each depth plane through each of the first homography transformation matrices to obtain each first target depth plane; Combining the first target depth planes according to the depth order to construct a first matching cost volume; Matching the first matching cost volume through a stereo convolutional neural network to obtain a first probability value that each region in the cost volume space belongs to a three-dimensional object; Normalizing the first probability value by a preset function, and weighted superimposing the normalized first probability value and its depth value to obtain the first target viewpoint depth map; If the resolution of the first target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the first target viewpoint depth map is determined as the final target viewpoint depth map.

4. The free viewpoint graph generation method according to claim 3, characterized in that: The preset group of scale feature maps of different resolutions of the multiple viewpoints includes a scale feature map with half resolution, After the first probability value is normalized by a preset function and the normalized first probability value and its depth value are weightedly superimposed to obtain the first target viewpoint depth map, the method further includes: If the resolution of the first target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the first target viewpoint depth map as a first initial value, and determining a first depth search range according to the first initial value and a first preset initial difference; Constructing a second homography transformation matrix in each depth plane according to the first depth search range and the internal and external parameters of the multiple viewpoints; Transforming the scale feature maps of the plurality of viewpoints at half the resolution to respective depth planes through the respective second homography transformation matrices to obtain respective second target depth planes; Combining each of the second target depth planes according to the depth order to construct a second matching cost volume; Matching the second matching cost volume by the stereo convolutional neural network to obtain a second probability value that each region in the cost volume space belongs to a three-dimensional object; Normalizing the second probability value by using the preset function, performing weighted superposition on the normalized second probability value and its depth value to obtain a second target viewpoint depth map; If the resolution of the second target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the second target viewpoint depth map is determined as the final target viewpoint depth map.

5. The free viewpoint graph generation method according to claim 4, characterized in that: The preset group of scale feature maps of different resolutions of the multiple viewpoints includes a scale feature map of double resolution, After normalizing the second probability value by the preset function and weighted superpositioning the normalized second probability value and its depth value to obtain a second target viewpoint depth map, the method further includes: If the resolution of the second target viewpoint depth map is smaller than the resolution of the multi-viewpoint map, taking the second target viewpoint depth map as a second initial value, and determining a second depth search range according to the second initial value and a second preset initial difference value; constructing a third homography transformation matrix in each depth plane according to the second depth search range and the internal and external parameters of the plurality of viewpoints; By using each of the third homography transformation matrices, the scale feature maps of the multiple viewpoints with one times the resolution are transformed to each depth plane to obtain each third target depth plane; Combining the third target depth planes according to the depth order to construct a third matching cost volume; Matching the third matching cost volume by using the stereo convolutional neural network to obtain a third probability value that each region in the cost volume space belongs to a three-dimensional object; Normalizing the third probability value by using the preset function, performing weighted superposition on the normalized third probability value and its depth value to obtain a third target viewpoint depth map; If the resolution of the third target viewpoint depth map is equal to the resolution of the multi-viewpoint map, the third target viewpoint depth map is determined as the final target viewpoint depth map.

6. The free viewpoint graph generation method according to claim 1, characterized in that: The step of extracting features from the multi-view image by a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image includes: Performing feature extraction on the multi-view image through a VGG convolutional neural network, and outputting feature images of the first three stages, wherein the sizes of the feature images of the first three stages are the original size, half size, and one quarter size of the multi-view image, respectively; Perform nearest neighbor interpolation on the half-size and quarter-size feature maps in the first three stages, and upsample them to a resolution consistent with the multi-view image; The processed half-size feature map and the processed quarter-size feature map are spliced ​​with the original-size feature map in the feature dimension to generate the depth coding maps to be processed.

7. The free viewpoint graph generation method according to claim 1, characterized in that: The method of projecting each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by the DIBR method to obtain a plurality of target depth coding images includes: In combination with the final target viewpoint depth map and its corresponding virtual camera parameters, each of the to-be-processed depth coding maps is projected to the target viewpoint through the DIBR method to obtain each of the target depth coding maps.

8. A free viewpoint map generating device, characterized in that: include: A first extraction module is used to extract features of the multi-view image by combining internal and external parameters of the multi-view image through a feature extraction network, and to obtain a final target view depth map by combining an unsupervised stereo matching network; A second extraction module is used to extract features from the multi-view image through a convolutional neural network to obtain a plurality of to-be-processed depth coding images of the multi-view image; A projection module, configured to project each of the to-be-processed depth coding images in combination with the final target viewpoint depth map by a DIBR method to obtain a plurality of target depth coding images; The fusion decoding module is used to fuse the target depth coding images through a preset aggregation module to obtain a target viewpoint coding image, and to decode the target viewpoint coding image through a fully convolutional network to obtain a target free viewpoint image.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the free viewpoint map generation method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the free viewpoint map generation method according to any one of claims 1 to 7 are implemented.

Images