Super-fractal multi-mode depth fusion-based virtual viewpoint occlusion area restoration method

Abstract

The invention discloses a super-fractal multi-modal depth fusion-based virtual viewpoint occlusion area restoration method, which belongs to the technical field of computer vision and image processing, and comprises the following steps of: performing multi-modal feature extraction on an input image by utilizing a constructed super-fractal multi-level feature fusion network; performing depth information estimation on the periphery of the occlusion region of the input image by using the constructed depth estimation model to obtain occlusion region filling content; selecting a current restoration scheme from given restoration schemes based on the obtained multi-modal features of the filling content of the occlusion area, and performing preliminary restoration on the occlusion area of the input image according to the weight of each modal feature in the current restoration scheme; and evaluating the preliminary restoration effect, and continuously optimizing the weight of each modal feature of the current restoration scheme when the restoration effect does not conform to the expectation until an image of which the restoration effect of the occlusion region meets the expectation is output. According to the invention, the quality and visual effect of the virtual viewpoint can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and specifically relates to a method for repairing a virtual viewpoint occluded area based on super fractal multimodal deep fusion. Background Art

[0002] In today's computer vision field, the technology of generating arbitrary perspective virtual viewpoints based on 2D images has a wide range of application scenarios. Whether in the immersive experience of virtual reality (VR) and augmented reality (AR), or in film and television production, game development, 3D reconstruction and other fields, this technology has a wide range of applications. However, the technology of generating arbitrary perspective virtual viewpoints based on 2D images still faces many challenges. Among them, the processing of occluded areas is one of the key factors affecting the quality of virtual viewpoints.

[0003] The challenges currently faced by the industry include the following: First, the inadequacy of depth estimation technology: When processing 2D images, most existing depth estimation models cannot effectively allocate attention according to the image semantic hierarchy and texture complexity, and cannot accurately capture the depth-related spatiotemporal features. Second, the problem of texture synthesis repair method: When dealing with occluded areas, the traditional texture synthesis repair method lacks effective evaluation of complex texture features in the method of judging texture authenticity, resulting in unsatisfactory repair effects. Third, the limitations of image repair technology: The partial differential equation model used in conventional image repair technology does not fully consider the biological visual mechanism and the complex physical properties of the image. Fourth, the dilemma of multi-image fusion repair: When using multiple images to fusion repair occluded areas, traditional image registration technology is difficult to fully utilize the multimodal information in the image. When processing images with different perspectives, illuminations, and complex spatiotemporal relationships, feature point matching is inaccurate, resulting in the inability to effectively fuse multi-image information for accurate repair. Fifth, the imperfection of repair effect evaluation: Existing repair effect evaluation methods are mainly based on simple visual features or limited manually designed indicators, and do not fully consider the neurobiological characteristics of the human visual system. Sixth, the selection of restoration technology lacks intelligence: when selecting restoration technology, the existing methods cannot dynamically optimize the selection of restoration methods according to actual conditions, and cannot meet the requirements of high-quality virtual viewpoint generation. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method for repairing occluded areas of virtual viewpoints based on super fractal multimodal deep fusion, which can improve the quality and visual effects of virtual viewpoints.

[0005] The present invention provides the following technical solutions:

[0006] The virtual viewpoint occlusion area repair method based on super fractal multimodal deep fusion includes:

[0007] The constructed hyper-fractal multi-level feature fusion network is used to extract multimodal features from the input image; the multimodal features include color features, semantic features, texture features and motion features;

[0008] The constructed depth estimation model is used to estimate the depth information around the occluded area of ​​the input image to obtain the filling content of the occluded area;

[0009] Based on the multimodal features of the obtained occluded area filling content, a current restoration scheme is selected from given restoration schemes, and the occluded area of ​​the input image is preliminarily restored according to the weights of each modal feature in the current restoration scheme; the given restoration schemes include a texture synthesis restoration scheme based on a super-fractal multimodal generative adversarial network, an image restoration scheme based on VAE and biological vision inspiration, and a multi-image registration restoration scheme based on multimodal fusion;

[0010] The effect of the preliminary restoration is evaluated, and when the restoration effect does not meet expectations, the weights of each modal feature of the current restoration scheme are continuously optimized until the output occluded area restoration effect meets the expected image.

[0011] Optionally, the hyperfractal multi-level feature fusion network includes:

[0012] A color feature extraction module is used to extract color features of an input image using a color space quantization algorithm;

[0013] The semantic feature enhancement module is used to enhance the semantic features of the extracted input image using a deep convolutional generative adversarial network;

[0014] A texture feature extraction module is used to extract texture features of an input image through an adaptive local image feature descriptor;

[0015] The motion feature extraction module is used to extract the motion features of the input image through the optical flow method and the particle filter algorithm;

[0016] The super fractal coding module is used to fuse the extracted color features, semantic features, texture features and motion features, and to mine the associations between different features using the self-similarity and recursive structure of the super fractal mathematical model.

[0017] Optionally, the structure of the depth estimation model is a fusion of a deep residual network and a long short-term memory network, and a hierarchical adaptive attention mechanism is introduced; the hierarchical adaptive attention mechanism is: obtaining the semantic hierarchy of the input image according to the extracted semantic features, and dynamically adjusting the weights of different semantic levels.

[0018] Optionally, the selecting a current repair solution from a given repair solution is specifically:

[0019] According to the extracted multimodal features, specific values ​​of the judgment indicators are obtained, and the specific values ​​of the judgment indicators are compared and selected with the data ranges of each judgment indicator in each restoration scheme; the judgment indicators include: the complexity of the texture around the occluded area, the semantic continuity of the boundary of the occluded area, the structural characteristics of natural objects and multi-image resources.

[0020] Optionally, in the texture synthesis restoration solution based on the super fractal multimodal generative adversarial network, the super fractal multimodal generative adversarial network includes: a generator network and a discriminator network;

[0021] The generator network uses superfractal mapping to initially fuse semantic features, color features, texture features and motion features of the occluded area boundary according to the weights of each modal feature, and generates a repair texture in the occluded area; the superfractal mapping is to map the features of different modalities into a unified superfractal space and maintain their respective feature information; when the generator network generates the repair texture in the occluded area, semantic coherence constraints and texture topology constraints are introduced;

[0022] The discriminator network discriminates the authenticity of the generated texture based on the super-fractal multimodal perception mechanism combined with texture topology perception;

[0023] The generator network and the discriminator network are adversarially trained through a super-fractal information interaction mechanism to optimize the restoration texture generated in the occluded area.

[0024] Optionally, the structure of the generator network is based on a multi-scale hyperfractal feature fusion generation architecture;

[0025] The generator network further fuses the features of each modality after preliminary fusion at different scales using superfractal transformation, and uses superfractal interpolation algorithm to generate repair texture in the occluded area;

[0026] The discriminator network adopts a multimodal superfractal fusion discriminant architecture; the discriminator network performs difference recognition between the generated repair texture and the original image texture through superfractal fusion at multiple resolutions, and simultaneously uses superfractal geometric metrics to evaluate the topological similarity between the generated repair texture and the original image texture.

[0027] Optionally, the image restoration solution based on VAE and biological vision inspiration specifically includes:

[0028] Step I: Establishing a physical constraint partial differential equation model inspired by biological vision, wherein the partial differential equation model is a hybrid model based on the combination of the optical wave equation, the heat conduction equation and the neural diffusion equation;

[0029] Step II: Use the VAE encoder to learn the latent semantic representation around the occluded area of ​​the input image;

[0030] Step III: Based on the weights of each modal feature, the partial differential equation model is used to guide the diffusion of latent semantic information from the periphery of the occluded region to the occluded region, so as to obtain the latent semantic representation of the occluded region;

[0031] Step IV: The VAE decoder is used to convert the latent semantic representation of the occluded area into the repaired image, and the repaired image is further corrected by using the biologically inspired error detection and repair network;

[0032] Step V: Repeat steps II to IV until a repaired image of the occluded area that meets the requirements is output.

[0033] Optionally, the multi-image registration and restoration solution based on multimodal fusion specifically includes:

[0034] Acquire multiple associated input images with different viewpoints, different lighting conditions, temporal and spatial correlations and containing scene semantic graphs related to the target input image;

[0035] Through the image registration method guided by multimodal feature matching based on the fusion of deep learning and graph neural network, the information of the non-occluded areas of the associated input image and the target input image is fused, and the occluded area of ​​the target input image is repaired according to the fused information.

[0036] Optionally, the image registration method guided by multimodal feature matching based on deep learning and graph neural network fusion includes:

[0037] According to the multimodal features and the weights of each modal feature, a multimodal convolution-based graph neural network is used to perform feature point matching to perform image registration between the associated input image and the target input image;

[0038] A transformation model based on 3D transformation invariance is used to estimate the 3D posture and shape changes of objects in all input images after image registration.

[0039] Optionally, the evaluation of the effect of the preliminary restoration includes: texture consistency between the restored area and the surrounding area at different frequencies, color consistency, visual smoothness, visual rationality based on retinal ganglion cell responses, and visual stability in complex scenes.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention extracts multimodal features through a constructed super-fractal multi-level feature fusion network, accurately determines the boundary information of the occluded area when performing depth estimation, and effectively improves the depth accuracy of the boundary of the hole area; the present invention can dynamically optimize the selection of the repair method according to the repair requirements in a given repair scheme, and iteratively optimize the repair results, thereby improving the accuracy and reliability of the repair and meeting the requirements for generating high-quality virtual viewpoints; in addition, the texture synthesis repair scheme based on the super-fractal multi-modal generative adversarial network of the present application utilizes the powerful generation ability of the generative adversarial network (GAN) and combines the super-fractal multi-modal features to synthesize the texture around the occluded area with high precision Consistent repair content; this high-precision texture synthesis helps to reduce the visual difference between the repaired image and the original image, and improves the naturalness and realism of the repair; in the image repair scheme based on VAE and biological vision inspiration of the present application, VAE can generate repair content that is coordinated with the surrounding content based on the contextual information around the occluded area, thereby improving the overall consistency and coherence of the repair. The use of biological vision inspiration helps to generate more natural, coherent and human visual expectations of the repair results; the multi-image registration repair scheme based on multimodal fusion of the present application can more comprehensively capture the details and features in the scene, and help generate more complete, accurate and rich repair content. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a virtual viewpoint occlusion area repair method based on super fractal multimodal deep fusion of the present invention;

[0043] Figure 2 It is a structural block diagram of the hyperfractal multi-level feature fusion network of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variation thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] Example 1

[0046] like Figure 1 As shown, a virtual viewpoint occlusion area repair method based on super fractal multimodal deep fusion is provided, comprising the following steps:

[0047] Step S1: extracting multimodal features from an input image using the constructed hyperfractal multi-level feature fusion network; the multimodal features include color features, semantic features, texture features, and motion features.

[0048] like Figure 2 As shown, the hyperfractal multi-level feature fusion network includes:

[0049] A color feature extraction module is used to extract color features of an input image using a color space quantization algorithm;

[0050] The semantic feature enhancement module is used to enhance the semantic features of the extracted input image using the deep convolutional generative adversarial network DCGAN;

[0051] A texture feature extraction module is used to extract texture features of an input image through an adaptive local image feature descriptor;

[0052] The motion feature extraction module is used to extract the motion features of the input image through the optical flow method and the particle filter algorithm;

[0053] The super fractal coding module is used to fuse the extracted color features, semantic features, texture features and motion features, and to mine the associations between different features using the self-similarity and recursive structure of the super fractal mathematical model.

[0054] Superfractal is a mathematical model with self-similarity and recursive structure. Self-similarity means that when observing an object at different scales, its local structure has similar characteristics to the overall structure; recursive structure means that complex forms are constructed by repeating certain rules or operations. For details, please refer to the existing technology. Superfractal can be used to analyze and process the characteristics of images. For example, the texture features in an image may have similar structural patterns at different scales. The self-similarity and recursive structure of superfractals can help to explore these potential relationships, so as to better understand and describe the image content.

[0055] The adversarial training of DCGAN can make the semantic features more representative and discriminative.

[0056] The adaptive local image feature descriptor can automatically adjust the description parameters according to the texture complexity of the local area of ​​the image to better capture the texture details. The commonly used adaptive local feature descriptor methods can be selected from the following methods: Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Local Binary Pattern (LBP) and its variants, and Histogram of Oriented Gradients (HOG).

[0057] The motion features are obtained by combining optical flow and particle filter analysis. Optical flow is used to estimate the direction and speed of the object, and particle filtering further optimizes the estimation of motion information, especially in the presence of noise and occlusion. These different modal features are fused through superfractal coding. The self-similarity of superfractals enables features at different scales to be correlated with each other, and the recursive structure mines the potential and deep relationship between different features, thereby more accurately determining the target object and possible occluded areas.

[0058] Step S2: Use the constructed depth estimation model to estimate the depth information around the occluded area of ​​the input image to obtain the filling content of the occluded area.

[0059] The structure of the depth estimation model is a fusion of the deep residual network ResNet and the long short-term memory network LSTM, and a hierarchical adaptive attention mechanism is introduced; the hierarchical adaptive attention mechanism is: obtaining the semantic level of the input image according to the extracted semantic features, and dynamically adjusting the weights of different semantic levels.

[0060] The semantic level of an image is divided into the visual level, the object level, and the conceptual level. Specifically, for images containing people and backgrounds, the people and backgrounds are first divided into different levels, and then further subdivided within each level according to the texture complexity. The attention weights of different regions are dynamically adjusted according to the level and complexity of these subdivisions, so that the model pays more attention to the areas that have an important impact on depth estimation. The fusion structure of the deep residual network (ResNet) and the long short-term memory network (LSTM) is used to extract depth-related spatiotemporal features. ResNet is good at extracting spatial features of images, while LSTM is used to process temporal information in image sequences. After the two are fused, the spatial changes of objects at different time points can be captured, so as to estimate depth information more accurately. The training data of the depth estimation model is obtained through a binocular stereo vision system combined with laser ranging and 2D image matching. This method can obtain more accurate depth labeling data. At the same time, zero-knowledge proof technology is used to ensure data security, and the legitimacy and integrity of the data are proved without leaking the data content during data transmission and storage.

[0061] The color features, semantic features, texture features, and motion features extracted in step S1 are saved in the form of matrices. By extracting features from the four matrices, the boundary information of the occluded area can be accurately determined. Then, based on this information, when performing depth estimation, the boundary of the occluded area will be imported as known information, thereby effectively improving the depth accuracy of the boundary of the hole area.

[0062] Step S3: Based on the multimodal features of the obtained occluded area filling content, select the current restoration scheme from the given restoration schemes, and perform preliminary restoration on the occluded area of ​​the input image according to the weights of each modal feature in the current restoration scheme; the given restoration schemes include a texture synthesis restoration scheme based on a super-fractal multimodal generative adversarial network, an image restoration scheme based on VAE and biological vision inspiration, and a multi-image registration restoration scheme based on multimodal fusion.

[0063] Select the current repair solution from the given repair solutions, specifically:

[0064] According to the extracted multimodal features, specific values ​​of the judgment indicators are obtained, and the specific values ​​of the judgment indicators are compared and selected with the data ranges of each judgment indicator in each restoration scheme; the judgment indicators include: the complexity of the texture around the occluded area, the semantic continuity of the boundary of the occluded area, the structural characteristics of natural objects and multi-image resources.

[0065] Furthermore, the texture complexity around the occluded area includes: texture entropy around the occluded area, texture contrast cross-coastal error, texture correlation and Gabor filter response amplitude variation coefficient.

[0066] Boundary semantic continuity includes: the transition ratio of pixels of different semantic categories on both sides of the boundary.

[0067] The structural features of natural objects include: the average value of contour curvature change and the number of skeleton branches.

[0068] When choosing a restoration method: when the texture around the occluded area is rich and complex, has dynamic features, or has high requirements for boundary semantic continuity, choose texture synthesis restoration based on super-fractal multimodal generative adversarial networks; when the restoration scene needs to simulate biological vision (such as medical image restoration), there are complex lighting and perspective changes, or when repairing the structure of natural objects, choose image restoration based on VAE and biological vision inspiration; if there are multiple images with different perspectives and lighting conditions, and it is necessary to use the scene semantic graph relationship or process the three-dimensional spatial changes of objects, choose multi-image registration restoration based on multimodal fusion.

[0069] When choosing a specific repair plan, the weight of each judgment indicator must also be considered.

[0070] As an option, the variable threshold ranges and the weights considered when selecting three given repair solutions are given:

[0071] (a) Texture synthesis restoration solution based on hyper-fractal multimodal generative adversarial network:

[0072] Texture complexity: Texture entropy greater than 3.0, texture contrast standard deviation greater than 0.25, texture correlation less than 0.4, Gabor filter response amplitude variation coefficient greater than 0.35. Weight consideration: 40%. High texture complexity is an important factor in choosing this method, because complex textures require accurate synthesis of repair textures through generative adversarial networks to match the surrounding area.

[0073] Boundary semantic continuity: The transition ratio of pixels of different semantic categories on both sides of the boundary is less than 0.15. Weight consideration: 30%. The high boundary semantic continuity requirement makes it more appropriate to use the hyper-fractal multimodal generative adversarial network to fuse boundary semantic information for restoration.

[0074] Natural object structure: The average value of the contour curvature change is greater than 0.15, and the number of skeleton branches is greater than 3 (a reference to the relative complexity of the natural object structure). Weight consideration: 20%. In the case of a certain degree of complexity in the natural object structure, this method can use the super-fractal characteristics to better handle the matching of the repaired texture and the object structure.

[0075] Multiple image resources: no special requirements (this method focuses on single image texture synthesis, if there are multiple images, it can assist in providing texture and other information). Weight consideration: 10%.

[0076] (b) Image restoration solution based on VAE and biological vision inspiration:

[0077] Texture complexity: Texture entropy is between 1.5-3.0, texture contrast standard deviation is between 0.15-0.25, texture correlation is between 0.3-0.4, Gabor filter response amplitude variation coefficient is between 0.2-0.35. Weight consideration: 20%. Moderate texture complexity is sufficient, because the focus is on biological vision-inspired restoration mechanisms, which rely relatively little on texture complexity.

[0078] Boundary semantic continuity: The transition ratio of pixels of different semantic categories on both sides of the boundary is between 0.1 and 0.2.

[0079] Weight consideration: 20%. The boundary semantic continuity requirement is relatively moderate, mainly combining biological vision mechanisms to handle repair.

[0080] Structural features of natural objects: the average value of the contour curvature change is between 0.1-0.15, and the number of skeleton branches is between 1-3. Weight consideration: 30%. For images with simple to moderately complex natural object structures, this method can better repair and restore when combined with the partial differential equations constrained by biological vision physics, especially when processing object structures in a simulated biological vision environment.

[0081] Multi-image resources: No special requirements (this method focuses on single-image combined with biological vision-inspired restoration). Weight consideration: 30%. Multi-image resources are not a core factor, but if there are auxiliary multi-images that provide information related to the biological vision of surrounding objects, the weight can be increased.

[0082] (c) Multi-image registration and restoration solution based on multimodal fusion:

[0083] Texture complexity: texture entropy less than 3.0, texture contrast standard deviation less than 0.25, texture correlation greater than 0.4, Gabor filter response amplitude variation coefficient less than 0.35. Weight consideration: 10%. When texture complexity is low, multi-image registration repair can better utilize non-texture information from other images to repair occluded areas.

[0084] Boundary semantic continuity: The transition ratio of pixels of different semantic categories on both sides of the boundary is greater than 0.15. Weight consideration: 20%. Relatively low boundary semantic continuity requirements, because multi-image registration can adjust the fusion of semantic information through other images.

[0085] Structural features of natural objects: the average value of the contour curvature change is less than 0.15, and the number of skeleton branches is less than 3. Weight consideration: 20%. For natural objects with simple structures, multi-image registration restoration can effectively use images from different perspectives for restoration without the need for complex natural object structure processing mechanisms.

[0086] Multi-image resources: at least 3 images with different viewing angles and lighting conditions, with the mutual information between images greater than 0.4. Weight consideration: 50%. Multi-image resources are the key factor in choosing this method. Sufficient multi-images can provide rich non-occluded area information for repairing occluded areas. The three repair methods of this application are:

[0087] 1. In the texture synthesis restoration scheme based on the super fractal multimodal generative adversarial network, the super fractal multimodal generative adversarial network includes: a generator network and a discriminator network.

[0088] The generator network uses super-fractal mapping to preliminarily fuse the semantic features, color features, texture features and motion features of the occluded area boundary according to the weights of each modal feature, and generates a repaired texture in the occluded area.

[0089] The hyperfractal mapping is as follows: features of different modes are mapped into a unified hyperfractal space, and their respective feature information is maintained; when the generator network generates restoration textures in occluded areas, semantic coherence constraints and texture topology constraints are introduced.

[0090] Introduce semantic coherence constraints to ensure that the generated texture is semantically consistent with the surrounding environment. For example, the object part in the repaired texture should be consistent with the category and attributes of the surrounding objects. Introduce texture topology constraints to ensure that the topology of the texture matches the surrounding textures to avoid unreasonable texture connections. The conditions of semantic coherence constraints and texture topology constraints can be determined based on expert experience.

[0091] The discriminator network judges the authenticity of the generated texture based on the hyperfractal multimodal perception mechanism combined with texture topology perception. The discriminator determines whether the texture meets the requirements by analyzing the feature distribution of the generated texture in the hyperfractal space and the similarity with the original image texture in multimodality and topology.

[0092] The generator network and the discriminator network are adversarially trained through a super-fractal information interaction mechanism to optimize the inpainted texture generated in the occluded area.

[0093] More specifically, the structure of the generator network is based on a multi-scale super-fractal feature fusion generation architecture; the generator network initially fuses the features of each modality at different scales, for example, at the micro scale, it fuses the detailed features of the texture and the local color changes; at the macro scale, it considers the semantic information and the overall motion trend; it uses the super-fractal transformation for further fusion, and at the same time uses the super-fractal interpolation algorithm to generate repaired textures in occluded areas. The super-fractal interpolation algorithm enhances the continuity of texture generation, so that the generated texture transitions naturally between different scales.

[0094] Super fractal transformation includes the steps of feature extraction, multi-scale iterative fusion (IFS), result generation and verification.

[0095] The discriminator network adopts a multimodal super-fractal fusion discriminant architecture; the discriminator network identifies the differences between the generated repair texture and the original image texture through super-fractal fusion at multiple resolutions, and uses super-fractal geometric metrics to evaluate the topological similarity of the generated repair texture and the original image texture, such as fractal dimension, similarity measure, etc., to evaluate the topological similarity of the texture, so as to judge the repair effect more comprehensively and accurately, and guide the generator network to generate more suitable repair textures.

[0096] The multimodal super-fractal fusion discrimination architecture includes a multimodal input layer, a super-fractal fusion module, and a discrimination decision layer. The working principle of the multimodal input layer is: the color, semantics, texture, motion and other modal features extracted from the image are simultaneously input into the architecture. These features are represented in the form of vectors or tensors. The features of different modalities are different in dimension and data meaning, but they all contain important information related to the image content. The fusion process of the super-fractal fusion module is as follows: in the super-fractal fusion module, multimodal features are fused at different resolutions; specifically, the features of different modalities are associated and integrated at different scales through super-fractal transformation; for example, the iterated function system (IFS) is used to represent the features of different modalities as functions, and feature fusion is achieved through multiple iterations; or the features of different modalities are adjusted according to the fractal dimension so that they can be better fused in the super-fractal space; in this process, the features at different resolutions complement each other to form a more comprehensive and representative fusion feature. The discriminant decision layer receives the fusion features output by the super-fractal fusion module, and evaluates the difference between the repaired texture and the original image texture by calculating the super-fractal geometric metrics. Based on these metric results, it determines whether the repaired texture is similar to the original image texture and whether the expected repair effect has been achieved. If the discriminant result shows that there is a large difference between the repaired texture and the original texture, it can be fed back to the generator network to guide it to adjust the generated repaired texture.

[0097] 2. Image restoration solution based on VAE and biological vision inspiration

[0098] The image restoration solution based on VAE and biological vision inspiration includes:

[0099] Step I: Establishing a physical constraint partial differential equation model inspired by biological vision, wherein the partial differential equation model is a hybrid model based on the combination of the optical wave equation, the heat conduction equation and the neural diffusion equation;

[0100] Step II: Use the VAE encoder to learn the latent semantic representation around the occluded area of ​​the input image; the latent semantic representation includes color features, semantic features, texture features and motion features;

[0101] Step III: Based on the weights of each modal feature, the partial differential equation model is used to guide the diffusion of latent semantic information from the periphery of the occluded region to the occluded region, so as to obtain the latent semantic representation of the occluded region;

[0102] Step IV: The VAE decoder is used to convert the latent semantic representation of the occluded area into the repaired image, and the repaired image is further corrected by using the biologically inspired error detection and repair network;

[0103] Step V: Repeat steps II to IV until a repaired image of the occluded area that meets the requirements is output.

[0104] Specifically, the VAE encoder is trained on a large amount of image data, and VAE maps the image to a low-dimensional latent space, in which the semantic content of the image is captured, such as the category and attributes of objects and the relationship between them. The structure of VAE refers to the existing technology.

[0105] The physical constraint partial differential equation model inspired by biological vision guides the diffusion of image information from around the occluded area to repair the occluded area based on the information processing mechanism of retinal ganglion cells and the principle of optical propagation. The physical constraint partial differential equation inspired by biological vision can refer to the existing technology. Retinal ganglion cells have a unique way of processing information, such as different responses to visual information of different directions and frequencies. Drawing on this mechanism, the partial differential equation can more accurately simulate the propagation and processing of image information in the visual system, while considering the propagation characteristics of light in different media, so that the repair process is more in line with the laws of biological visual perception. The two work together to optimize the quality of repair, and at the same time introduce a neural plasticity adjustment mechanism to adjust the repair parameters according to the feedback information in the repair process to improve the adaptability of the repair. The partial differential equation is a hybrid model based on the optical wave equation, the heat conduction equation and the neural diffusion equation. The optical wave equation describes the propagation process of light in the medium, taking into account the wave nature of light and the refraction and reflection characteristics of the medium to light. The heat conduction equation simulates the information diffusion process in the image similar to heat transfer, for example, it plays a role in processing the smooth transition of color and brightness information. The neural diffusion equation is based on the conduction characteristics of neural signals in the retina and optic nerves, and is used to simulate the processing and transmission of visual information in the biological visual system. This hybrid model can more accurately describe the diffusion law of image information in complex occlusion scenes. Under different lighting conditions, the optical wave equation plays a major role to ensure that the propagation of light conforms to the laws of physics; when processing the local smoothness of the image, the heat conduction equation optimizes the distribution of color and brightness; when considering biological visual perception, the neural diffusion equation makes the repaired image more consistent with the processing results of the human visual system. At the same time, the repair error is corrected using a biologically inspired error detection and repair network, which simulates the error detection and repair mechanism in the biological nervous system to improve the accuracy of the repair.

[0106] The biologically inspired error detection and repair network includes an input layer, a feature extraction layer, an error detection layer, an error location module, a repair strategy generation layer, a repair layer and an output layer. The repaired image first enters the input layer, extracts features through the feature extraction layer, the error detection layer identifies errors, the error location module determines the error location, the repair strategy generation layer generates a repair strategy, the repair layer repairs the error location according to the strategy, and finally outputs the further repaired image through the output layer. Through continuous learning and optimization, the accuracy of error detection and repair is improved, so that the repaired image is closer to the real unobstructed image. The structure of each layer can refer to the prior art. As an option, the error detection layer and the error location module can be implemented based on a convolutional neural network (CNN), the repair strategy generation layer can be implemented based on a convolutional neural network or a reinforcement learning model, and the repair layer can be implemented using an existing image repair network.

[0107] 3. Multi-image registration and restoration solution based on multimodal fusion, specifically including:

[0108] Acquire multiple associated input images with different perspectives, different lighting conditions, spatiotemporal correlation and containing scene semantic graphs related to the target input image; fuse the information of the non-occluded areas of the associated input images and the target input image through an image registration method guided by multimodal feature matching based on the fusion of deep learning and graph neural networks, and repair the occluded areas of the target input image based on the fused information.

[0109] Image registration methods guided by multimodal feature matching based on the fusion of deep learning and graph neural network include:

[0110] According to the features of each modal feature and the weight of each modal feature, a graph neural network based on multimodal convolution is used to perform feature point matching to perform image registration between the associated input image and the target input image; a transformation model based on three-dimensional transformation invariance is used to estimate the three-dimensional posture change and shape change of all input images after image registration.

[0111] When repairing the occluded area, multiple 2D images with different perspectives, different lighting conditions, spatiotemporal correlation and scene semantic graphs are combined. The corresponding non-occluded area information in multiple images is fused to repair the occluded area in the target 2D image through the image registration technology guided by multimodal feature matching based on deep learning and graph neural network fusion. The multimodal feature matching guidance mechanism uses the semantic, texture, color, motion clues in the image and the relationship information of the scene semantic graph to improve the registration accuracy. The scene semantic graph provides high-level semantic relationships between objects and scenes in the image, such as the relative position and action relationship between objects. By combining this information, it is possible to find corresponding relationships in different images more accurately, and accurate registration can be achieved even under complex perspective and lighting changes. The multimodal convolutional graph neural network can effectively capture the multimodal features and graph structure information of objects in the image for accurate matching of feature points. It extracts local features of the image through convolution operations, and uses graph neural networks to process semantic relationships and topological structures in the image. The combination of the two can more accurately find feature points of the same object in different images. The method based on three-dimensional transformation invariance can adapt to the shape changes of objects and posture changes in three-dimensional space to accurately estimate the transformation model. By analyzing the geometric relationship and posture change rules of objects in three-dimensional space, the transformation relationship between images can be accurately calculated regardless of how the objects are rotated, translated or deformed, thereby achieving more accurate image registration.

[0112] Step S4: Evaluate the effect of the preliminary restoration, and when the restoration effect does not meet expectations, continuously optimize the weights of each modal feature of the current restoration scheme until an image with a restoration effect that meets expectations is output.

[0113] The effect of the preliminary restoration was evaluated, including: texture consistency between the restored area and the surrounding area at different frequencies, color consistency, visual smoothness, visual rationality based on the response of retinal ganglion cells, and visual stability in complex scenes.

[0114] The method of evaluating the preliminary restoration effect can refer to the existing technology. As an option, it can be jointly evaluated based on the human eye visual perception model, deep learning classifier and multimodal biologically inspired similarity evaluation index.

[0115] The human visual perception model simulates the neurobiological characteristics of the human visual system, including the response characteristics of retinal cells, the conduction characteristics of the visual nerves, and the processing mechanism of the visual cortex of the brain. For example, consider the differences in sensitivity of different areas of the retina to different frequencies and brightness, and the way the visual nerves process different directions and motion information. The deep learning classifier is trained with a large amount of annotated restoration effect data to learn image feature patterns under different restoration qualities for classification and evaluation of restoration effects. Multimodal bio-inspired similarity evaluation indicators include texture consistency, color consistency, visual smoothness, visual rationality based on the response of retinal ganglion cells, and visual stability in complex scenes between the restoration area and the surrounding area at different frequencies. These indicators comprehensively consider the multimodal characteristics of the image and biological visual perception factors to more comprehensively measure the restoration quality.

[0116] The initial weight of each modal feature in the repair scheme can be determined based on expert experience. When the weight of each modal feature of the current repair scheme is continuously adjusted, the adjustment can be performed according to a set rule or a set adjustment method.

[0117] Example 2

[0118] A specific restoration example is given: virtual viewpoint generation and occlusion processing for indoor scenes. Consider a 2D image sequence of an indoor living room scene. There are objects such as sofas, coffee tables, TVs, and people in the image. There are different degrees of occlusion, such as people being partially occluded by furniture and objects occluding each other.

[0119] First, we collected 2D images of indoor scenes from multiple angles and under different lighting conditions. We used a binocular stereo vision system and a laser ranging device to obtain the depth information of some images, accurately matched the depth information with the corresponding 2D images, and manually annotated the scene semantic graph to mark the positional relationship (such as the sofa in front of the coffee table) and functional relationship (such as a person sitting on the sofa watching TV) between objects. We preprocessed the images, normalized them to the [0,1] interval, cropped out irrelevant edge parts, uniformly scaled them to a suitable size (such as 512×512 pixels), and used Gaussian filtering to remove possible noise.

[0120] Then, through the super-fractal multi-level feature fusion network, the color features extracted by the CIELAB color space quantization method, the semantic features generated by DCGAN through adversarial training, the texture features obtained by the adaptive local binary pattern (LBP) descriptor, and the motion features analyzed by the optical flow method combined with the particle filter, the super-fractal encoder sets the number of iterations to 10 and the fractal dimension to 2.5 to accurately determine the target object (sofa, person, etc.) and the occluded area (such as the part of the person occluded by the sofa). The depth estimation model based on the hierarchical adaptive attention mechanism and ResNet-LSTM fusion is used to accurately estimate its depth information and clarify the depth information of the occluded area (such as the occluded part of the person). The training data is obtained through a binocular stereo vision system combined with laser ranging and 2D image matching, and the zero-knowledge proof technology is used to ensure data security.

[0121] Secondly, the restoration technology is dynamically selected according to the semantic content, geometric shape, depth information, position in the image, interaction relationship of surrounding objects, multimodal features and scene semantic graph information of the occluded area. For the part of the character occluded by the sofa, a texture synthesis restoration method based on super fractal multimodal generative adversarial network (SMM-GAN) is used. For the situation where the texture of some sofas is blurred due to illumination changes (similar to the occlusion effect), an image restoration restoration technology based on variational autoencoder (VAE) coupled with physical constraint partial differential equations inspired by biological vision is used. Assuming that there are multiple indoor scene images from different perspectives, the corresponding non-occluded area information in multiple images is fused through the image registration technology guided by multimodal feature matching based on deep learning and graph neural network fusion to repair the occluded area in the target image.

[0122] Finally, the restoration effect is evaluated and output. For the restored image, the restoration effect is jointly evaluated using a human visual perception model, a deep learning classifier, and a multimodal biologically inspired similarity evaluation index, including texture consistency between the repaired area and the surrounding area at different frequencies (such as the consistency between the repaired texture of the character's clothing and the texture of the surrounding clothing), color consistency (the coordination between the color of the repaired area and the color of the surrounding environment), visual smoothness (smooth transition of the repaired boundary), visual rationality based on the response of retinal ganglion cells (such as whether the repaired texture conforms to the visual perception of the human eye to the object), and visual stability in complex scenes (such as the stability of the restoration effect under different lighting and viewing angle changes). The restoration process is optimized according to the evaluation results until a satisfactory effect is achieved, and finally a high-quality virtual viewpoint image of the indoor scene is output.

[0123] Example 3

[0124] A restoration example is given: virtual viewpoint generation and occlusion processing for outdoor traffic scenes. For a 2D image of a busy urban traffic intersection scene, including cars, pedestrians, traffic lights, road signs and other objects, there are a lot of occlusions, such as occlusions between cars and pedestrians being blocked by cars or traffic facilities.

[0125] First, we collected 2D image datasets of traffic intersections at different times, weather conditions (sunny, rainy, cloudy, etc.) and shooting angles. We used a binocular stereo vision system and a laser ranging device to obtain the depth information of some images, matched the depth information with the 2D images, and automatically generated a scene semantic map through a semantic segmentation algorithm. We marked the categories of objects (cars, pedestrians, traffic lights, etc.) and the relationships between them (such as cars driving on the road, pedestrians on the sidewalk, etc.). We preprocessed the images, normalized them to the range of [0,1], cropped the sky or building parts that were irrelevant to the edge of the image, and uniformly scaled them to a suitable size (such as 1024×768 pixels). We also used a median filter to remove salt and pepper noise from the images.

[0126] Then, through the super-fractal multi-level feature fusion network, the color features extracted by the HSV color space quantization method, the semantic features generated by DCGAN through adversarial training, the texture features obtained by the adaptive Gabor filter, and the motion features analyzed by the optical flow method combined with the particle filter, the super-fractal encoder sets the number of iterations to 12 and the fractal dimension to 2.8 to accurately determine the target objects (cars, pedestrians, etc.) and occluded areas (such as occluded pedestrians or car parts). The depth estimation model based on the hierarchical adaptive attention mechanism and ResNet-LSTM fusion is used to accurately estimate its depth information and determine the depth information of the occluded area (such as the part of the car body occluded by other cars). The training data is obtained through a binocular stereo vision system combined with laser ranging and 2D image matching, and the zero-knowledge proof technology is used to ensure data security.

[0127] Secondly, the restoration technology is dynamically selected based on the semantic content, geometric shape, depth information, position in the image, interaction relationship of surrounding objects, multimodal features and scene semantic graph information of the occluded area. For partially occluded car bodies, a texture synthesis restoration method based on super fractal multimodal generative adversarial network (SMM-GAN) is used. For the situation where traffic lights are partially blurred due to rainy days (similar to occlusion effects), an image restoration restoration technology based on variational autoencoder (VAE) coupled with physical constraint partial differential equations inspired by biological vision is used. If there are multiple traffic intersection images from different perspectives, the corresponding non-occluded area information in multiple images is fused through image registration technology guided by multimodal feature matching based on deep learning and graph neural network fusion to repair the occluded area in the target image.

[0128] Finally, the repaired image is evaluated using a human visual perception model, deep learning classifier, and multimodal bio-inspired similarity evaluation index, including texture consistency between the repaired area and the surrounding area at different frequencies (such as the consistency between the repaired texture of the car body and the surrounding body texture), color consistency (the coordination between the color of the repaired area and the surrounding environment), visual smoothness (smooth transition of the repaired boundary), visual rationality based on the response of retinal ganglion cells (such as whether the repaired texture conforms to the visual perception of traffic objects by the human eye), and visual stability in complex scenes (such as the stability of the repair effect under different weather and viewing angle changes). The repair process is optimized according to the evaluation results until a satisfactory effect is achieved, and finally a high-quality virtual viewpoint image of the outdoor traffic scene is output.

[0129] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system, device, and storage medium disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0130] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0131] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A virtual viewpoint occlusion area repair method based on super fractal multimodal deep fusion, characterized in that: include: The constructed hyper-fractal multi-level feature fusion network is used to extract multimodal features from the input image; the multimodal features include color features, semantic features, texture features and motion features; The constructed depth estimation model is used to estimate the depth information around the occluded area of ​​the input image to obtain the filling content of the occluded area; Based on the multimodal features of the obtained occluded area filling content, a current restoration scheme is selected from given restoration schemes, and the occluded area of ​​the input image is preliminarily restored according to the weights of each modal feature in the current restoration scheme; the given restoration schemes include a texture synthesis restoration scheme based on a super-fractal multimodal generative adversarial network, an image restoration scheme based on VAE and biological vision inspiration, and a multi-image registration restoration scheme based on multimodal fusion; The effect of the preliminary restoration is evaluated, and when the restoration effect does not meet expectations, the weights of each modal feature of the current restoration scheme are continuously optimized until the output occluded area restoration effect meets the expected image.

2. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1 is characterized in that: The hyperfractal multi-level feature fusion network includes: A color feature extraction module is used to extract color features of an input image using a color space quantization algorithm; The semantic feature enhancement module is used to enhance the semantic features of the extracted input image using a deep convolutional generative adversarial network; A texture feature extraction module is used to extract texture features of an input image through an adaptive local image feature descriptor; The motion feature extraction module is used to extract the motion features of the input image by using the optical flow method and the particle filter algorithm; The super fractal coding module is used to fuse the extracted color features, semantic features, texture features and motion features, and to mine the associations between different features using the self-similarity and recursive structure of the super fractal mathematical model.

3. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1, characterized in that: The structure of the depth estimation model is a fusion of a deep residual network and a long short-term memory network, and a hierarchical adaptive attention mechanism is introduced; the hierarchical adaptive attention mechanism is: obtaining the semantic hierarchy of the input image according to the extracted semantic features, and dynamically adjusting the weights of different semantic hierarchies.

4. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1, characterized in that: The current repair scheme is selected from the given repair schemes, specifically: According to the extracted multimodal features, specific values ​​of the judgment indicators are obtained, and the specific values ​​of the judgment indicators are compared and selected with the data ranges of each judgment indicator in each restoration scheme; the judgment indicators include: the complexity of the texture around the occluded area, the semantic continuity of the boundary of the occluded area, the structural characteristics of natural objects and multi-image resources.

5. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1, characterized in that: In the texture synthesis restoration scheme based on the super fractal multimodal generative adversarial network, the super fractal multimodal generative adversarial network includes: a generator network and a discriminator network; The generator network uses superfractal mapping to initially fuse semantic features, color features, texture features and motion features of the occluded area boundary according to the weights of each modal feature, and generates a repair texture in the occluded area; the superfractal mapping is to map the features of different modalities into a unified superfractal space and maintain their respective feature information; when the generator network generates the repair texture in the occluded area, semantic coherence constraints and texture topology constraints are introduced; The discriminator network discriminates the authenticity of the generated texture based on the super-fractal multimodal perception mechanism combined with texture topology perception; The generator network and the discriminator network are adversarially trained through a super-fractal information interaction mechanism to optimize the restoration texture generated in the occluded area.

6. The method for repairing virtual viewpoint occlusion areas based on super fractal multimodal depth fusion according to claim 5, characterized in that: The structure of the generator network is based on a multi-scale hyperfractal feature fusion generation architecture; The generator network further fuses the features of each modality after preliminary fusion at different scales using superfractal transformation, and uses superfractal interpolation algorithm to generate repair texture in the occluded area; The discriminator network adopts a multi-modal super-fractal fusion discriminant architecture; The discriminator network identifies the difference between the generated repair texture and the original image texture through superfractal fusion at multiple resolutions, and evaluates the topological similarity between the generated repair texture and the original image texture using superfractal geometry metrics.

7. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1, characterized in that: The image restoration solution based on VAE and biological vision inspiration specifically includes: Step I: Establishing a physical constraint partial differential equation model inspired by biological vision, wherein the partial differential equation model is a hybrid model based on the combination of the optical wave equation, the heat conduction equation and the neural diffusion equation; Step II: Use the VAE encoder to learn the latent semantic representation around the occluded area of ​​the input image; Step III: Based on the weights of each modal feature, the partial differential equation model is used to guide the diffusion of latent semantic information from the periphery of the occluded region to the occluded region, so as to obtain the latent semantic representation of the occluded region; Step IV: The VAE decoder is used to convert the latent semantic representation of the occluded area into the repaired image, and the repaired image is further corrected by using the biologically inspired error detection and repair network; Step V: Repeat steps II to IV until a repaired image of the occluded area that meets the requirements is output.

8. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1, characterized in that: The multi-image registration and restoration solution based on multimodal fusion specifically includes: Acquire multiple associated input images with different viewpoints, different lighting conditions, temporal and spatial correlations and containing scene semantic graphs related to the target input image; Through the image registration method guided by multimodal feature matching based on the fusion of deep learning and graph neural network, the information of the non-occluded areas of the associated input image and the target input image is fused, and the occluded area of ​​the target input image is repaired according to the fused information.

9. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 8, characterized in that: The image registration method guided by multimodal feature matching based on deep learning and graph neural network fusion includes: According to the multimodal features and the weights of each modal feature, a multimodal convolution-based graph neural network is used to perform feature point matching to perform image registration between the associated input image and the target input image; A transformation model based on 3D transformation invariance is used to estimate the 3D posture and shape changes of objects in all input images after image registration.

10. The method for repairing virtual viewpoint occlusion area based on super fractal multimodal depth fusion according to claim 1, characterized in that: The effect of the preliminary restoration is evaluated, including: texture consistency between the restored area and the surrounding area at different frequencies, color consistency, visual smoothness, visual rationality based on retinal ganglion cell responses, and visual stability in complex scenes.

Images