Training method and training device of image generation model and electronic equipment

Abstract

The embodiment of the invention discloses a training method and training device of an image generation model and electronic equipment. A specific implementation mode of the method comprises the steps that a disparity map of a sample image pair is determined according to image data in the sample image pair, and the sample image pair comprises a first sample view and a second sample view which are different in view angle; performing image transformation on the first sample view according to the disparity map to obtain an initial second view; a pre-constructed image generation model is adopted, multiple filling processing is conducted on the initial second view, the second view is generated according to multiple filling images obtained through filling, and the image generation model comprises at least two filling modules different in filling method; and according to a comparison result of the second view and the second sample view, adjusting model parameters of the image generation model so as to continue model training until training is completed. The embodiment is related to a three-dimensional image processing technology, and the filling effect of the image can be improved through the filling image obtained by fusing multiple filling methods.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of three-dimensional image processing, and in particular to a training method, a training device and an electronic device for an image generation model. Background Art

[0002] With the continuous development of electronic products, immersive 3D content experience is becoming increasingly popular. However, a large amount of existing online digital media (such as video clips, TV series and movies) is still in 2D form, so there is an urgent need to convert 2D videos into 3D videos.

[0003] However, the inventors found that the image filling methods in related video conversion methods, such as interpolation filling or network filling based on deep learning, have some problems and cannot achieve a good three-dimensional conversion effect.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a training method for an image generation model, a training device for an image generation model, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for training an image generation model, comprising: determining a disparity map of a sample image pair based on image data in a sample image pair, wherein the sample image pair includes a first sample view and a second sample view with different perspectives; performing an image transformation on the first sample view based on the disparity map to obtain an initial second view; using a pre-built image generation model to perform multiple filling processes on the initial second view, and generating a second view based on multiple filled images obtained by filling, wherein the image generation model includes at least two filling modules with different filling methods; adjusting model parameters of the image generation model based on a comparison result between the second view and the second sample view to continue model training until the training is completed.

[0008] In some embodiments, a pre-built image generation model is used to perform multiple filling processes on the initial second view, including: inputting the initial second view into a first filling module in the image generation model, using a polygon filling algorithm to perform filling processing, and obtaining a first filled image.

[0009] In some embodiments, a pre-built image generation model is used to perform multiple filling processes on the initial second view, which also includes: inputting the initial second view into a second filling module in the image generation model, and using a deep learning-based video editing method to perform filling processing to obtain a second filled image.

[0010] In some embodiments, a pre-built image generation model is used to perform multiple filling processes on the initial second view, which also includes: inputting the initial second view into a third filling module of the image generation model, extending and filling the foreground edge in the initial second view, and obtaining a third filled image.

[0011] In some embodiments, the foreground edge in the initial second view is expanded and filled, including: using an edge detection algorithm to determine the edge position of the disparity map, and determining the foreground pixels and background pixels at the edge position; replacing a specified number of background pixels located in a target direction of the edge position with foreground pixels adjacent thereto to obtain a replacement disparity map; and filling the initial second view according to the replacement disparity map.

[0012] In some embodiments, a second view is generated based on multiple filled images obtained by filling, including: using a fusion module in an image generation model to perform image feature processing on the multiple filled images to obtain a color image and a mask image corresponding to each filled image; fusing the color image and the multiple mask images to generate a second view and output it.

[0013] In some embodiments, a color image and a plurality of mask images are fused to generate and output a second view, including: determining a first coefficient based on two fill images among a plurality of fill images and a mask image corresponding to one of the two fill images; determining a second coefficient based on the first coefficient, a mask image corresponding to the other of the two fill images, and the remaining fill images among the plurality of fill images; generating a second view of the first sample view at another viewing angle based on the second coefficient, the color image, and the remaining mask image, and outputting the second view.

[0014] In some embodiments, the fusion module includes multiple encoding layers and corresponding multiple decoding layers; and image feature processing is performed on multiple filled images, including: for each encoding layer in the multiple encoding layers, convolutionally fusing the input features of the encoding layer with the relevant features of the next adjacent encoding layer to obtain the output features of the encoding layer, wherein the relevant features are features obtained by upsampling the input features; and jump-connecting the output features of the encoding layer with the corresponding decoding layer to achieve feature fusion.

[0015] In some embodiments, according to the comparison result between the second view and the second sample view, the model parameters of the image generation model are adjusted, including: according to the second view and the second sample view, using multiple loss functions to respectively determine multiple loss values, wherein the multiple loss functions include at least two loss functions with different calculation methods; determining the model loss value of the image generation model according to the multiple loss values, and adjusting the model parameters according to the model loss value.

[0016] In a second aspect, some embodiments of the present disclosure provide a three-dimensional video generation method, including: performing depth prediction on a video image in a two-dimensional video to obtain a disparity map of the video image; performing image transformation on the video image based on the disparity map of the video image to obtain an initial second view; inputting the initial second view into an image generation model to generate a second view of the video image, wherein the image generation model is trained using the training method of the image generation model described in any implementation method of the first aspect above; and generating a three-dimensional video of the two-dimensional video based on the video image and the corresponding second view.

[0017] In a third aspect, some embodiments of the present disclosure provide a training device for an image generation model, comprising: a disparity map determination unit, configured to determine a disparity map of a sample image pair based on image data in the sample image pair, wherein the sample image pair includes a first sample view and a second sample view with different perspectives; an image transformation unit, configured to perform image transformation on the first sample view based on the disparity map to obtain an initial second view; a view generation unit, configured to use a pre-built image generation model to perform multiple filling processes on the initial second view, and generate a second view based on multiple filled images obtained by filling, wherein the image generation model includes at least two filling modules with different filling methods; a model adjustment unit, configured to adjust model parameters of the image generation model based on a comparison result between the second view and the second sample view, so as to continue model training until the training is completed.

[0018] In some embodiments, the view generation unit includes a first filling subunit configured to input the initial second view into a first filling module in the image generation model, perform filling processing using a polygon filling algorithm, and obtain a first filled image.

[0019] In some embodiments, the view generation unit further includes a second filling subunit configured to input the initial second view into a second filling module in the image generation model, perform filling processing using a deep learning-based video editing method, and obtain a second filled image.

[0020] In some embodiments, the view generation unit further includes a third filling subunit configured to input the initial second view into a third filling module of the image generation model, and perform extended filling on the foreground edge in the initial second view to obtain a third filled image.

[0021] In some embodiments, the third filling subunit is further configured to use an edge detection algorithm to determine the edge position of the disparity map, and to determine the foreground pixels and background pixels at the edge position; replace a specified number of background pixels located in the target direction of the edge position with foreground pixels adjacent thereto to obtain a replacement disparity map; and fill the initial second view according to the replacement disparity map.

[0022] In some embodiments, the view generation unit also includes a fusion subunit, which is configured to use a fusion module in an image generation model to perform image feature processing on multiple fill images to obtain a color image and a mask image corresponding to each fill image; perform fusion processing on the color image and multiple mask images to generate a second view and output it.

[0023] In some embodiments, the fusion subunit is further configured to determine a first coefficient based on two fill images among a plurality of fill images and a mask map corresponding to one of the two fill images; determine a second coefficient based on the first coefficient, a mask map corresponding to the other of the two fill images, and the remaining fill images among the plurality of fill images; generate a second view of the first sample view at another perspective based on the second coefficient, the color image, and the remaining mask map, and output the second view.

[0024] In some embodiments, the fusion module includes multiple encoding layers and corresponding multiple decoding layers; and the fusion sub-unit is further configured to, for each encoding layer in the multiple encoding layers, perform convolution fusion on the input features of the encoding layer with the relevant features of the next adjacent encoding layer to obtain the output features of the encoding layer, wherein the relevant features are features obtained by upsampling the input features; and jump-connect the output features of the encoding layer to the corresponding decoding layer to achieve feature fusion.

[0025] In some embodiments, the model adjustment unit is further configured to use multiple loss functions to respectively determine multiple loss values ​​according to the second view and the second sample view, wherein the multiple loss functions include at least two loss functions with different calculation methods; determine the model loss value of the image generation model according to the multiple loss values, and adjust the model parameters according to the model loss value.

[0026] In a fourth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the image generation model described in any implementation method in the first aspect above.

[0027] In a fifth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the training method of the image generation model described in any one of the implementations in the first aspect above is implemented.

[0028] In a sixth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the training method of the image generation model described in any implementation manner in the first aspect above.

[0029] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the training method of the image generation model of some embodiments of the present disclosure can improve the generation effect of images from different perspectives. Specifically, in the field of computer vision, there are relatively few related technical solutions for converting two-dimensional to three-dimensional, and these technologies usually use a single filling method. These filling methods all have some problems. For example, the interpolation method is prone to pixel stretching and obvious burrs at the edge of the foreground and background; and the filling method based on deep learning may cause instability (jitter) between frames during video processing.

[0030] Based on this, in the training method of the image generation model of some embodiments of the present disclosure, after performing image transformation on the first sample view according to the disparity map of the sample image pair to obtain the initial second view, multiple (at least two) filling modules in the pre-built image generation model can be used to perform multiple filling processes on the initial second view, thereby obtaining multiple filled images. Since the filling methods of these filling modules are different, filled images with different filling effects can be obtained. In this way, based on multiple filled images, the second view corresponding to the first sample view can be generated, and the filling results of each filling method can be optimized and fused, thereby obtaining a filling result that is better than a single filling method. In this way, the filling effect of the image can be improved, and a more robust and natural filling result can be obtained, thereby improving the quality of the image generated by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0032] Figure 1 is a flowchart of some embodiments of the training method of the image generation model disclosed in the present invention;

[0033] Figure 2 are flowcharts of other embodiments of the training method of the image generation model disclosed in the present invention;

[0034] Figure 3 yes Figure 2 Flowcharts of some embodiments of a feature update unit FUU;

[0035] Figure 4 It is a schematic diagram of the structure of some embodiments of the training device of the image generation model disclosed in the present invention;

[0036] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0038] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0039] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0040] In addition, the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0041] Figure 1A process 100 of some embodiments of a method for training an image generation model according to the present disclosure is shown. The method may include the following steps:

[0042] Step 101: determine a disparity map of a sample image pair according to image data in the sample image pair.

[0043] In some embodiments, the execution subject (such as a server) of the training method of the image generation model disclosed in the present invention can be connected to other electronic devices (such as user terminals, databases, etc.) through wired or wireless connections. Here, the user can send a sample image data set for training the model to the execution subject through the user terminal. Alternatively, the user can store the sample image data set in a database and send the storage location (such as a database identifier) ​​to the execution subject. In this way, the execution subject can obtain the sample image data set from it.

[0044] It should be noted that the generation of three-dimensional images usually requires two-dimensional images of multiple (at least two) perspectives to be realized, so the above-mentioned sample image data set usually contains paired sample images, that is, sample image pairs. Among them, the sample image pair may include a first sample view and a second sample view with different perspectives. The perspective here can be set according to actual needs, such as a top-down perspective and an upward perspective, usually a left perspective and a right perspective. In this way, the model can be trained to predict the output of another perspective view based on a certain input perspective view. For example, if the first sample view (or the second sample view) is input, the second sample view (or the first sample view) is predicted and output.

[0045] In some embodiments, the execution subject may first determine the disparity map of the sample image pair, that is, the disparity map between the first sample view and the second sample view, based on the image data in the sample image pair. In the field of computer vision, a disparity map is usually an image based on any image in the image pair, whose size is the size of the reference image and whose element values ​​are disparity values. As an example, the execution subject may use the first sample view as a reference and adopt the NCC (normalization cross-correlation) method to determine the disparity map from the first sample view to the second sample view.

[0046] Optionally, the execution subject may also use a stereo matching algorithm to estimate depth information (such as Figure 2 DepthEstimation in the image processing unit is used to obtain the disparity map corresponding to the first sample view. It can be understood that the stereo matching algorithm is an important problem in computer vision, which aims to find matching corresponding points from images of different viewpoints, and is usually used to generate a disparity map to estimate the depth information of the scene.

[0047] In some embodiments, the sample image dataset may be obtained from public resources or acquired through collection. As an example, an image acquisition device (such as a mobile phone, camera, etc.) may be used to shoot spatial videos, which may then be processed by software and code to obtain side-by-side videos from different perspectives to form high-quality training and evaluation datasets. Here, the dataset may include hundreds (such as 500) videos and more than a thousand images with high-definition resolution (such as 1180*1180), including various real scenes, such as indoors, outdoors, in shopping malls, parks, etc.

[0048] Step 102: Perform image transformation on the first sample view according to the disparity map to obtain an initial second view.

[0049] In some embodiments, the execution subject may perform image transformation (such as Figure 2 The initial second view is usually generated by using the depth-guided warp in the sample image pair. The initial second view here is usually the same as the second sample view in the sample image pair. It should be noted that the view initially generated based on the depth information, that is, the initial second view, usually needs to be filled with the hole area to obtain the final second view. Therefore, the initial second view here usually also includes filling mask data, such as Figure 2 Occlusion Masks.

[0050] It is understandable that Depth-Guided warp usually uses depth information to guide the warping operation of the image. Warp is usually an image processing technique used to change the geometric shape of an image. By remapping the pixel positions of the image, image warping can achieve various deformation effects of the image, such as stretching, compression, rotation, translation, etc. This technology is widely used in computer vision, image editing and special effects production.

[0051] Step 103 , using a pre-built image generation model, performing a variety of filling processes on the initial second view, and generating a second view based on a plurality of filled images obtained by filling.

[0052] In some embodiments, for the initial second view obtained in step 102, the execution subject may use a pre-built image generation model to perform multiple filling processes on the initial second view, thereby obtaining multiple filled images. The image generation model includes at least two filling modules with different filling methods. The filling method can be set according to actual needs.

[0053] As an example, the initial second view can be input into the first filling module in the image generation model, such as Figure 2In the poly-base, a polygon filling algorithm is used to fill the image and obtain the first filled image (such as Figure 2 middle ). Here. Polygon filling generally refers to coloring the pixels within a polygonal area to form a continuous shape in computer graphics. Polygon filling can be achieved through a variety of algorithms, such as seed filling, boundary filling algorithm and scan line filling algorithm.

[0054] In some embodiments, the polygon filling algorithm can treat the rows of the image as polylines, and generate and deform these polylines by applying parallax adjustment before rasterization. First, each edge of the polygon is determined and scanned in the vertical direction. For each row (or scan line), the intersection of the row and the edge of the polygon is calculated. Then, the intersections of each row are sorted according to the horizontal position (x coordinate), and these intersections define the visible part of the polygon on the row. Next, the intersections on each row are connected into line segments, which constitute the projection of the polygon on the row and are regarded as part of the polyline. After that, deformation is performed based on the disparity map, and the deformed line segments are converted into pixel data. Finally, pixels are filled between the intersections of each row. It should be noted that this filling method can ensure the stability of the video and reduce the jitter of the subtitles, but it is easy to cause pixel stretching and a large number of burrs for the edge filling area.

[0055] Optionally, the initial second view can be input into a second filling module in the image generation model, such as Figure 2 In DE-base, a deep learning-based video editing method is used to perform filling processing to obtain the second filled image (such as Figure 2 middle ). The deep learning-based video editing method here can also be set according to actual needs, such as using artificial intelligence (AI) video editing tools such as ComfyUI-MochiEdit or Propainte.

[0056] It is understandable that the above-mentioned AI video editing tools (such as Propainte) have usually been trained on a large amount of data, so they have good generalization. The tool can propagate information on both adjacent images and features through optical flow, and use known pixels in other frames to fill the area that needs to be filled. Then the remaining unfilled areas are filled through the designed filling network. This filling method can achieve better filling effects. However, since it is based on a deep learning network, it cannot guarantee the stability of the video and is prone to generating shaky subtitles.

[0057] Furthermore, the execution entity may generate a second view based on the multiple filled images obtained by filling. As an example, the execution entity may perform feature fusion on the image data of the multiple filled images, thereby using the filled image after feature fusion as the generated second view. For example, for the edge area of ​​the filled image obtained by filling with the polygon filling algorithm, the video editing method based on deep learning may be fused to fill the image data at the edge of the filled image, thereby improving the filling effect of the edge filled area and reducing or avoiding the occurrence of pixel stretching and a large number of burrs. For another example, for the entire image, the image data in the filled image obtained by filling with the video editing method based on deep learning may be used, so as to ensure the filling effect of the entire image. For the text part in the image, the image data in the filled image obtained by filling with the polygon filling algorithm may be used, so as to ensure the stability of the image.

[0058] It is understandable that there are usually some problems in the relevant filling methods. For example, the traditional interpolation method is prone to pixel stretching and obvious burrs at the edge of the foreground and background; and the filling method based on deep learning may cause instability (jitter) between frames during video processing. The method of the embodiment of the present disclosure can use a variety of filling methods to fill the initial second view, thereby obtaining a plurality of filled images with different filling effects. Then, based on these filled images, a second view corresponding to the first sample view is generated, which can combine the advantages of different filling methods and improve the filling effect of the image.

[0059] Step 104: adjust the model parameters of the image generation model according to the comparison result between the second view and the second sample view, so as to continue the model training until the training is completed.

[0060] In some embodiments, the execution subject may compare and analyze the second view generated in step 103 with the second sample view in the sample image pair corresponding to the first sample view. Thus, according to the comparison result of the two, the model parameters of the image generation model are adjusted to continue training the adjusted model until the training is completed. For example, according to the second view and the second sample view (such as Figure 2 GT Right Frames), to determine the value of the loss function (such as Figure 2 The loss function here can also be set according to actual needs, such as similarity.

[0061] In some application scenarios, in order to improve the learning and training effect of the model, multiple loss values ​​can be determined by using multiple loss functions according to the generated second view and the second sample view. The multiple loss functions can include at least two loss functions with different calculation methods. The model loss value of the image generation model can be determined according to the multiple loss values, such as weighted summing the multiple loss values. The model parameters can be adjusted according to the model loss value.

[0062] As an example, the method of the embodiment of the present disclosure may adopt three different loss functions, namely content loss, perceptual loss, and adversarial loss. Among them, the content loss may calculate the loss value between the generated second view and the real second view (i.e., the second sample view), which may be expressed by the following formula:

[0063]

[0064] Among them, I g represents the corresponding real second view; Represents the generated second view.

[0065] For the perceptual loss, the generated second view and the real second view can be simultaneously input into a pre-trained network model, such as VGG19 (a deep convolutional neural network architecture), and the output features of the middle layer are taken to calculate it, which can be expressed by the following formula:

[0066]

[0067] Among them, φ() represents the convolutional network; j represents the corresponding level.

[0068] For adversarial loss, it can be obtained by training a discriminator. Usually, the discriminator needs to distinguish the generated images from the real images as much as possible, while the generator needs to generate real fake images as much as possible to confuse the discriminator. It can be expressed by the following formula:

[0069]

[0070] Where D represents the discriminator; P g represents the distribution of generated data; P data Represents the distribution of real data.

[0071] From the above description, it can be seen that the training method of the image generation model of some embodiments of the present disclosure, after performing image transformation on the first sample view according to the disparity map of the sample image pair to obtain the initial second view, can use multiple (at least two) filling modules in the pre-constructed image generation model to perform multiple filling processes on the initial second view, thereby obtaining multiple filled images. Since the filling methods of these filling modules are different, filled images with different filling effects can be obtained. In this way, based on multiple filled images, the second view corresponding to the first sample view can be generated, and the filling results of each filling method can be optimized and fused, thereby obtaining a filling result that is better than a single filling method. In this way, the filling effect of the image can be improved, and a more robust and natural filling result can be obtained, thereby improving the quality of the image generated by the model.

[0072] Continue to see Figure 2 , which shows a flowchart of other embodiments of the training method of the image generation model disclosed in the present invention.

[0073] In the embodiment of the present disclosure, a multi-branch filling fusion framework is proposed in the filling stage. That is, the image generation model of the present disclosure may include three filling modules, such as Figure 2 As shown in FIG. 1 , they are respectively a first filling module for polygon filling, a second filling module based on deep learning, and a third filling module based on parallax expansion (DL-base). That is, after obtaining the initial second view, it can be input into the three filling modules of the model respectively. The filling process of the first and second filling modules can be referred to in Figure 1 In addition, the initial second view may be input into the third filling module of the image generation model to extend and fill the foreground edge in the initial second view, thereby obtaining a third filling image (such as Figure 2 middle ).

[0074] Specifically, first, an edge detection algorithm (such as Canny) can be used to determine the edge position of the disparity map, as well as the foreground pixels and background pixels at the edge position. Here, it can be judged according to the size of the pixel value at the edge position. For example, pixels with relatively large pixel values ​​(or greater than a set value) can be determined as foreground pixels; and pixels with relatively small pixel values ​​(or less than a set value) can be determined as background pixels. Then, a specified number of background pixels located in the target direction of the edge position can be replaced with adjacent foreground pixels to obtain a replaced disparity map. For example, two background pixels in the horizontal direction at the edge position can be directly replaced with the values ​​of adjacent foreground pixels.

[0075] It can be understood that if the right view is generated, then usually only the right side of the edge can be focused on. In the right edge of the right view, the background pixels are generally located on the right side of the edge, and the foreground pixels are generally located on the left side of the edge. In this case, the values ​​of the two adjacent background pixels on the right side of the edge can be replaced with the values ​​of the foreground pixels on the left side of the edge that are adjacent to the edge. If the left view is generated, then usually only the left side of the edge can be focused on. In the left edge of the left view, the background pixels are generally located on the left side of the edge, and the foreground pixels are generally located on the right side of the edge. In this case, the values ​​of the two adjacent background pixels on the left side of the edge can be replaced with the values ​​of the foreground pixels on the right side of the edge that are adjacent to the edge.

[0076] Afterwards, the initial second view can be filled according to the replacement disparity map. It should be noted that although the filling method based on the deep learning network can obtain a better filling effect, this method is also prone to foreground penetration. In the embodiment of the present disclosure, since the side edge contour in the replacement disparity map is the foreground image, and the side edge contour of the original image (the first sample view and the initial second view) is the background image, background information can be provided for the foreground edge, so that the filling module of the model can correctly fill the background content. In other words, by expanding the edge of the foreground and background of the disparity map toward the background direction, additional background information is added to the original foreground edge, so that the subsequent filling module can refer to the background information to avoid foreground penetration. In this way, the phenomenon of pixel stretching and foreground penetration that occurs in other filling methods can be improved, and the filling effect can be further improved. It should be noted that this method may provide incorrect background reference pixels in areas with complex backgrounds.

[0077] Optionally, the initial second view may be copied, and an edge detection algorithm may be used to determine the foreground edge of the copied initial second view. In addition, a specified number (e.g., 1) of foreground pixels in the target direction at the foreground edge of the copied initial second view may be replaced with background pixels adjacent thereto to obtain a third view. In this way, the original initial second view may be filled according to the third view. For the original initial second view and the filled images obtained by other filling methods, background information may also be provided at the foreground edge, thereby improving the filling effect of the foreground edge.

[0078] Furthermore, if Figure 2 As shown, the method of the embodiment of the present disclosure also proposes a fusion module for hierarchical feature updating, which is used to integrate the advantages of multiple filling methods. Specifically, the fusion module in the image generation model can be used to perform image feature processing on multiple filling images to obtain a color image (such as Figure 2 RGB content map C), and the mask maps corresponding to each filled image (such as Figure 2Then, the color image and the multiple mask images are fused to generate and output the second view (such as Figure 2 middle ).

[0079] As an example, first, according to two filling images among the plurality of filling images and a mask image corresponding to one of the two filling images, the first coefficient may be determined. Next, the second coefficient may be determined according to the first coefficient, the mask image corresponding to the other of the two filling images, and the remaining filling images of the plurality of filling images. Afterwards, according to the second coefficient, the color image, and the remaining mask image, a second view of the first sample view at another viewing angle can be generated and output. See the following formula:

[0080]

[0081] It should be noted that the structure of the above fusion module can also be set according to actual needs. For example, the fusion module may include multiple encoding layers and corresponding multiple decoding layers. Among them, the encoding layer is usually used for encoding processing of the image; and the decoding layer is opposite to the encoding layer and is usually used for decoding processing of the image.

[0082] Furthermore, in the structure of the fusion module, Figure 2 As shown, a feature update unit (FUU) is also proposed in the embodiment of the present disclosure. The FUU unit can update and fuse the coding layer features of different resolutions, and finally add the fused features to the decoding layer features of the corresponding resolution to complete information fusion.

[0083] Specifically, Figure 3 As shown, for each coding layer in the multiple coding layers in the fusion module, in addition to encoding the input features, the FUU unit convolutionally fuses the input features of the coding layer with the relevant features of the next adjacent coding layer to obtain the output features of the coding layer. The relevant features are the features obtained by upsampling the input features. And the output features of the coding layer are jump-connected to the corresponding decoding layer to achieve feature fusion. See the following formula:

[0084]

[0085] Among them, Conv represents the convolutional layer in the FUU unit; f i m Represents the input features of the mth encoding layer; represents the features obtained by upsampling the input features of the m+1th encoding layer; f o mIt represents the final fused feature output of the mth encoding layer and jumps to the decoding layer of the corresponding resolution.

[0086] From the above description, it can be seen that the training method of the image generation model of the embodiment of the present disclosure further enriches and improves the image filling and image fusion process. That is, a disparity expansion filling strategy is further proposed. By extending the edge of the disparity map, the effect of the subsequent filling module and the accuracy of background content filling can be effectively improved. In addition, a fusion module for hierarchical feature iterative update is also proposed. Through the designed FUU unit, multi-resolution features are effectively updated and fused. By generating the corresponding attention mask, the filling results of each branch are effectively fused, and detailed content optimization is provided by generating a content map.

[0087] In some application scenarios, when the image generation model of the present disclosure is trained, the model can be used to generate a view different from the input view angle, thereby converting a two-dimensional image into a three-dimensional image. Here, some embodiments of the present disclosure also propose a three-dimensional video generation method. The method may include the following steps:

[0088] First, a video image (such as a left view) in a two-dimensional video may be depth predicted to obtain a disparity map of the video image. The two-dimensional video here may be input by a user or obtained from other electronic devices according to instructions.

[0089] Next, the video image may be transformed according to the disparity map of the video image to obtain an initial second view (such as a right view).

[0090] Next, the initial second view may be input into the image generation model, thereby generating the second view of the video image. The image generation model may be trained using the image generation model training method described in any implementation in the above embodiment.

[0091] Afterwards, a 3D video of the 2D video can be generated based on the video image and the corresponding second view. For example, the video image of the 2D video and the corresponding second view can be alternately displayed on a video playback device to form a 3D video effect.

[0092] That is to say, after completing the training of the image generation model disclosed in the present invention, we will obtain a model that can complete the function of converting 2D to 3D video. In this way, spatial videos can be viewed on various smart devices (AR / VR) according to views from different perspectives (such as left and right views).

[0093] Further references Figure 4 , as a response to the above Figures 1 to 3The present disclosure provides some embodiments of a training device for an image generation model. Figures 1 to 3 The device can be specifically applied to various electronic devices.

[0094] like Figure 4 As shown, the training device 400 of the image generation model of some embodiments may include: a disparity map determination unit 401, configured to determine the disparity map of the sample image pair according to the image data in the sample image pair, wherein the sample image pair includes a first sample view and a second sample view with different viewing angles; an image transformation unit 402, configured to perform image transformation on the first sample view according to the disparity map to obtain an initial second view; a view generation unit 403, configured to use a pre-built image generation model to perform multiple filling processes on the initial second view, and generate a second view based on multiple filling images obtained by filling, wherein the image generation model includes at least two filling modules with different filling methods; a model adjustment unit 404, configured to adjust the model parameters of the image generation model according to the comparison result between the second view and the second sample view, so as to continue the model training until the training is completed.

[0095] In some embodiments, the view generation unit 403 may include a first filling subunit (not shown in the figure), which is configured to input the initial second view into the first filling module in the image generation model, use a polygon filling algorithm to perform filling processing, and obtain a first filled image.

[0096] In some embodiments, the view generation unit 403 may further include a second filling subunit (not shown in the figure), which is configured to input the initial second view into a second filling module in the image generation model, and use a deep learning-based video editing method to perform filling processing to obtain a second filled image.

[0097] In some embodiments, the view generation unit 403 may further include a third filling subunit (not shown in the figure), which is configured to input the initial second view into the third filling module of the image generation model, expand and fill the foreground edge in the initial second view, and obtain a third filled image.

[0098] In some embodiments, the third filling subunit can be further configured to use an edge detection algorithm to determine the edge position of the disparity map, and determine the foreground pixels and background pixels at the edge position; replace a specified number of background pixels located in the target direction of the edge position with foreground pixels adjacent thereto to obtain a replacement disparity map; and fill the initial second view according to the replacement disparity map.

[0099] In some embodiments, the view generation unit 403 may further include a fusion subunit (not shown in the figure), which is configured to use a fusion module in the image generation model to perform image feature processing on multiple filling images to obtain a color image and a mask image corresponding to each filling image; perform fusion processing on the color image and the multiple mask images to generate a second view and output it.

[0100] In some embodiments, the fusion subunit can be further configured to determine a first coefficient based on two fill images among a plurality of fill images and a mask map corresponding to one of the two fill images; determine a second coefficient based on the first coefficient, a mask map corresponding to the other of the two fill images, and the remaining fill images among the plurality of fill images; generate a second view of the first sample view at another perspective based on the second coefficient, the color image, and the remaining mask map, and output the second view.

[0101] In some embodiments, the fusion module may include multiple encoding layers and corresponding multiple decoding layers; and the fusion sub-unit may be further configured to, for each encoding layer in the multiple encoding layers, perform convolution fusion on the input features of the encoding layer with the relevant features of the next adjacent encoding layer to obtain the output features of the encoding layer, wherein the relevant features are features obtained by upsampling the input features; and jump-connect the output features of the encoding layer to the corresponding decoding layer to achieve feature fusion.

[0102] In some embodiments, the model adjustment unit 404 can be further configured to use multiple loss functions to respectively determine multiple loss values ​​according to the second view and the second sample view, wherein the multiple loss functions include at least two loss functions with different calculation methods; determine the model loss value of the image generation model according to the multiple loss values, and adjust the model parameters according to the model loss value.

[0103] It can be understood that the units recorded in the training device 400 of the image generation model are similar to the reference Figures 1 to 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the training device 400 of the image generation model and the units contained therein, and will not be described in detail here.

[0104] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0105] like Figure 5As shown, the electronic device 500 may include a processing device 501 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0106] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a speaker, a vibrator, etc.; storage devices 508 including, for example, a disk, a hard disk, a memory card, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0107] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0108] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0109] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0110] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines the disparity map of the sample image pair according to the image data in the sample image pair, wherein the sample image pair includes a first sample view and a second sample view with different viewing angles; performs image transformation on the first sample view according to the disparity map to obtain an initial second view; uses a pre-constructed image generation model to perform multiple filling processes on the initial second view, and generates a second view according to multiple filling images obtained by filling, wherein the image generation model includes at least two filling modules with different filling methods; adjusts the model parameters of the image generation model according to the comparison result between the second view and the second sample view to continue model training until the training is completed.

[0111] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The units described may also be arranged in a processor, for example, may be described as: a processor includes a disparity map determination unit, an image transformation unit, a view generation unit, and a model adjustment unit. The names of these units do not constitute limitations on the units themselves in certain cases, for example, the disparity map determination unit may also be described as a "unit for determining a disparity map of a sample image pair".

[0114] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned training methods for the image generation model.

[0116] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A training method for an image generation model, comprising: Determine a disparity map of the sample image pair according to image data in the sample image pair, wherein the sample image pair includes a first sample view and a second sample view having different viewing angles; Performing image transformation on the first sample view according to the disparity map to obtain an initial second view; Using a pre-built image generation model, performing multiple filling processes on the initial second view, and generating a second view according to multiple filled images obtained by filling, wherein the image generation model includes at least two filling modules with different filling methods; According to the comparison result between the second view and the second sample view, the model parameters of the image generation model are adjusted to continue the model training until the training is completed.

2. The training method of the image generation model according to claim 1, wherein: The pre-built image generation model is used to perform multiple filling processes on the initial second view, including: The initial second view is input into a first filling module in the image generation model, and a polygon filling algorithm is used to perform filling processing to obtain a first filled image.

3. The training method of the image generation model according to claim 1, wherein: The method of using a pre-built image generation model to perform multiple filling processes on the initial second view further includes: The initial second view is input into a second filling module in the image generation model, and a filling process is performed using a deep learning-based video editing method to obtain a second filled image.

4. The method for training an image generation model according to claim 1, wherein: The method of using a pre-built image generation model to perform multiple filling processes on the initial second view further includes: The initial second view is input into a third filling module of an image generation model, and the foreground edge in the initial second view is extended and filled to obtain a third filled image.

5. The method for training an image generation model according to claim 4, wherein: The step of extending and filling the foreground edge in the initial second view comprises: Using an edge detection algorithm to determine the edge position of the disparity map, and to determine the foreground pixels and the background pixels at the edge position; Replacing a specified number of background pixels located in a target direction of the edge position with adjacent foreground pixels to obtain a replacement disparity map; The initial second view is filled according to the replacement disparity map.

6. The method for training an image generation model according to claim 1, wherein: Generating a second view according to the multiple filled images obtained by filling includes: Using the fusion module in the image generation model, image feature processing is performed on multiple filled images to obtain a color image and a mask image corresponding to each filled image; The color image and the plurality of mask images are fused to generate a second view and output it.

7. The method for training an image generation model according to claim 6, wherein: The step of fusing the color image and the plurality of mask images to generate and output a second view comprises: Determine a first coefficient according to two filling images among the plurality of filling images and a mask image corresponding to one of the two filling images; determining a second coefficient according to the first coefficient, a mask image corresponding to another of the two filling images, and the remaining filling images of the plurality of filling images; A second view of the first sample view at another viewing angle is generated according to the second coefficient, the color image, and the remaining mask image, and the second view is output.

8. The method for training an image generation model according to claim 6, wherein: The fusion module includes a plurality of encoding layers and a corresponding plurality of decoding layers; as well as The performing image feature processing on the multiple filled images comprises: For each coding layer in the multiple coding layers, convolutionally fuse the input features of the coding layer with the related features of the next adjacent coding layer to obtain the output features of the coding layer, wherein the related features are features obtained by upsampling the input features; The output features of the encoding layer are jump-connected to the corresponding decoding layer to achieve feature fusion.

9. The method for training an image generation model according to claim 1, wherein: The adjusting the model parameters of the image generation model according to the comparison result between the second view and the second sample view includes: According to the second view and the second sample view, using multiple loss functions to respectively determine multiple loss values, wherein the multiple loss functions include at least two loss functions with different calculation methods; A model loss value of the image generation model is determined according to the multiple loss values, and a model parameter is adjusted according to the model loss value.

10. A three-dimensional video generation method, comprising: Performing depth prediction on a video image in a two-dimensional video to obtain a disparity map of the video image; According to the disparity map of the video image, performing image transformation on the video image to obtain an initial second view; Inputting the initial second view into an image generation model to generate a second view of the video image, wherein the image generation model is trained using the image generation model training method according to any one of claims 1 to 9; A three-dimensional video of the two-dimensional video is generated according to the video image and the corresponding second view.

11. A training device for an image generation model, comprising: a disparity map determining unit, configured to determine a disparity map of the sample image pair according to image data in the sample image pair, wherein the sample image pair includes a first sample view and a second sample view having different viewing angles; an image transformation unit, configured to perform image transformation on the first sample view according to the disparity map to obtain an initial second view; a view generation unit configured to use a pre-built image generation model to perform multiple filling processes on the initial second view, and to generate a second view according to multiple filled images obtained by filling, wherein the image generation model includes at least two filling modules with different filling methods; The model adjustment unit is configured to adjust the model parameters of the image generation model according to the comparison result between the second view and the second sample view, so as to continue the model training until the training is completed.

12. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the image generation model as described in any one of claims 1-9.

13. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, it implements the training method of the image generation model as described in any one of claims 1-9.

14. A computer program product, comprising a computer program, which, when executed by a processor, implements the training method of the image generation model as described in any one of claims 1 to 9.

Images