Street scene reconstruction method, electronic equipment and storage medium

Abstract

The invention discloses a street scene reconstruction method, device and system and a storage medium. The reconstruction method comprises the following steps: acquiring a target image set shot for a target street scene; reconstructing a target street scene based on the target street scene model according to the obtained target image set; wherein the target street scene model is trained based on a first data set and a second data set, and the first data set is constructed based on a training image set shot for the target street scene and is used for training a three-dimensional model to obtain an initial street scene model; the second data set is constructed based on an unseen view angle image set and a shot training image set, and the unseen view angle image set is obtained by using the obtained initial street scene model. As the used target street scene model is obtained through iterative optimization of the initial street scene model, geometric reconstruction precision is improved when the target street scene model is used for street scene reconstruction.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a street scene reconstruction method, electronic equipment and storage medium. Background Art

[0002] Street scene geometry reconstruction has important application value in the fields of computer vision and robotics, and is widely used in autonomous driving simulation, virtual reality, augmented reality, urban planning, and map updates. These applications require high-precision 3D reconstruction of real-world street scenes for accurate simulation and interaction in virtual environments.

[0003] Related technologies often use COLMAP for 3D reconstruction. COLMAP is a widely used SfM and MVS tool that uses robust feature detection and matching algorithms to reconstruct sparse and dense point clouds of scenes from multiple images.

[0004] However, since street scenes are often blocked by dynamic or static objects such as vehicles, pedestrians and vegetation, some areas cannot be captured by the camera, resulting in incomplete data when COLMAP performs three-dimensional reconstruction, affecting the reconstruction accuracy. Summary of the invention

[0005] Based on the above-mentioned defects and shortcomings of the prior art, the present application proposes a street scene reconstruction method, electronic device and storage medium, which significantly improve the accuracy of street scene geometric reconstruction through iterative optimization.

[0006] According to a first aspect of the present application, a method for reconstructing a street scene is provided, comprising:

[0007] Acquire a target image set shot for a target street scene;

[0008] Reconstructing the target street scene based on the obtained target image set and the target street scene model;

[0009] The target street scene model is pre-trained based on a first data set and a second data set. The first data set is constructed based on a training image set shot for the target street scene and is used to train a pre-established three-dimensional model to obtain an initial street scene model. The second data set is constructed based on an unseen perspective image set and a shot training image set. The unseen perspective image set is obtained using the obtained initial street scene model.

[0010] According to the street scene reconstruction method provided in the first aspect of the present application,

[0011] The second dataset is constructed in the following way:

[0012] Performing image enhancement processing on partially occluded images in the obtained unseen view image set to obtain a first image set; constructing the second data set based on the first image set and the captured training image set;

[0013] or,

[0014] Perform image enhancement processing on partially occluded images in the obtained unseen view image set to obtain a first image set; perform image enhancement processing on images in non-Lambert lighting areas and areas lacking texture features in the captured training image set to obtain a second image set; and construct the second data set based on the first image set and the second image set.

[0015] According to the street scene reconstruction method provided in the first aspect of the present application,

[0016] In the case where the second data set is constructed based on the first image set and the captured training image set, the image enhancement process is performed on the partially occluded images in the obtained unseen view image set to obtain the first image set, including:

[0017] Detecting whether each image in the set of unseen viewing angle images has an artifact region, and masking the artifact region therein;

[0018] The masked images are subjected to image completion processing, and a set of unseen viewpoint images in which the images have undergone image completion processing is used as the first image set.

[0019] According to the street scene reconstruction method provided in the first aspect of the present application,

[0020] In the case where the second data set is constructed based on the first image set and the second image set, performing image enhancement processing on the partially occluded images in the obtained unseen view image set to obtain the first image set includes:

[0021] Detecting whether each image in the set of unseen viewing angle images has an artifact region, and masking the artifact region therein;

[0022] Performing image completion processing on the masked images, and taking a set of unseen viewpoint images in which the images have been subjected to the image completion processing as the first image set;

[0023] The step of performing image enhancement processing on images in the captured training image set having non-Lambertian illumination areas and areas lacking texture features to obtain a second image set comprises:

[0024] Detecting whether there is an artifact region in each image in the captured image set, and masking the artifact region therein;

[0025] The masked image is subjected to image completion processing, and a set of test images obtained by shooting and in which the image is subjected to image completion processing is used as the second image set.

[0026] According to the street scene reconstruction method provided in the first aspect of the present application, the image completion processing of the masked image includes:

[0027] Calculate the degree of mask uncertainty for each masked image;

[0028] The two-dimensional diffusion model is used to perform image completion processing on the mask-processed image whose mask uncertainty degree is greater than a first preset threshold.

[0029] According to the street scene reconstruction method provided in the first aspect of the present application, the first data set includes: a COLMAP data set, and the second data set includes: a COLMAP data set;

[0030] The three-dimensional model includes: a three-dimensional model represented by a 3D-GS method;

[0031] The two-dimensional diffusion model includes: a two-dimensional diffusion model trained in a DDPM sampling manner.

[0032] According to the street scene reconstruction method provided by the first aspect of the present application, the first data set is obtained by:

[0033] Performing image preprocessing on the images in the training image set;

[0034] Extracting feature points from each image that has undergone image preprocessing, performing feature point matching between different images that have undergone image preprocessing based on the extracted feature points and using a bidirectional nearest neighbor matching method, and obtaining multiple pairs of matching images with feature point matching pairs;

[0035] Estimate the initial pose of the camera using the obtained multiple pairs of matching images;

[0036] Sparse reconstruction is performed based on the obtained initial pose and feature point matching pairs to obtain other poses other than the initial pose of the camera and a three-dimensional point cloud for reconstructing the street scene;

[0037] Minimize the reprojection error for all camera poses and the obtained 3D point cloud bundles;

[0038] The first data set is constructed according to the camera's intrinsic parameters, the processed posture and the processed three-dimensional point cloud, and the matching information of multiple pairs of matching images.

[0039] According to the street scene reconstruction method provided in the first aspect of the present application, the captured training image set includes: an image sequence in which the overlapping area between adjacent images is not less than a second preset threshold.

[0040] According to a second aspect of the present application, there is provided an electronic device, including: a memory and a processor;

[0041] The memory is connected to the processor and is used to store programs;

[0042] The processor is used to implement any of the above-mentioned methods for reconstructing street scenes by running the program in the memory.

[0043] According to a third aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for reconstructing a street scene as described above is implemented.

[0044] The present application provides a street scene reconstruction method, device and storage medium, which obtains a target image set shot for a target street scene; reconstructs the target street scene based on the obtained target image set and a target street scene model; wherein the target street scene model is pre-trained based on a first data set and a second data set, the first data set is constructed based on a training image set shot for the target street scene, and is used to train a pre-established three-dimensional model to obtain an initial street scene model; the second data set is constructed based on an unseen perspective image set and a shot training image set, and the unseen perspective image set is obtained using the obtained initial street scene model. Since the target street scene model used is obtained through iterative optimization of the initial street scene model, the geometric reconstruction accuracy of the street scene is improved when it is used for street scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of a flow chart of a method for reconstructing a street scene provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a process flow of line image completion processing provided in an embodiment of the present application;

[0048] Figure 3A schematic diagram of a process for constructing a first data set for reconstructing a street scene provided in an embodiment of the present application;

[0049] Figure 4 A schematic flow chart of another method for reconstructing a street scene provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a street scene reconstruction process provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a street scene reconstruction device provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The present application embodiment provides a method for reconstructing a street scene, such as Figure 1 As shown, the method includes:

[0055] Step 100: Obtain a target image set shot for a target street scene;

[0056] Step 110: reconstruct a target street scene according to the obtained target image set and based on a target street scene model; wherein the target street scene model is pre-trained based on a first data set and a second data set, the first data set is constructed based on a training image set shot for the target street scene, and is used to train a pre-established three-dimensional model to obtain an initial street scene model; the second data set is constructed based on an unseen viewpoint image set and a shot training image set, and the unseen viewpoint image set is obtained using the obtained initial street scene model.

[0057] In the embodiments of the present application, the unseen viewing angle refers to the camera viewing angle that is not included in the initial training process, which is usually located outside or at an extreme angle of the training viewing angle and cannot be directly obtained from the training data.

[0058] Exemplarily, images in the target image set shot for the target street scene and the training image set shot for the target street scene may both be RGB images.

[0059] Exemplarily, the pre-established three-dimensional model may include: a three-dimensional model represented by the 3D-GS method, NeRF (Neural Radiance Fields), and Instant-NGP (Instant-Neural Graphic Primitives). Among them, the three-dimensional model represented by the 3D-GS method is a neural radiation field model based on Gaussian point cloud, which is used for high-quality three-dimensional reconstruction and image rendering. The training goal of minimizing the difference between the rendered image and the real image is achieved through a set loss function. The set loss function may include: mean square error and adversarial loss. The training uses back propagation and optimization algorithms to optimize the model parameters to obtain the final trained model.

[0060] In the embodiment of the present application, the training image set refers to an image set used to train a three-dimensional model.

[0061] The street scene reconstruction method provided in the embodiment of the present application obtains a target image set shot for a target street scene; reconstructs the target street scene based on the obtained target image set and a target street scene model; wherein the target street scene model is pre-trained based on a first data set and a second data set, the first data set is constructed based on a training image set shot for the target street scene, and is used to train a pre-established three-dimensional model to obtain an initial street scene model; the second data set is constructed based on an unseen perspective image set and a shot training image set, and the unseen perspective image set is obtained using the obtained initial street scene model. Since the target street scene model used is obtained through iterative optimization of the initial street scene model, the reconstruction quality of the model is improved, and therefore when it is used to reconstruct the street scene, the geometric reconstruction accuracy of the street scene is improved.

[0062] In an exemplary embodiment, the second data set is constructed by:

[0063] Firstly, image enhancement processing is performed on the partially occluded images in the obtained unseen view image set to obtain a first image set;

[0064] Secondly, the second data set is constructed based on the first image set and the captured training image set.

[0065] The construction process of the first data set and the second data set is similar, the difference is that the construction of the first data set is based on a set of captured images, and the construction of the second data set is based on a set of generated unseen viewpoint images and a set of captured images.

[0066] The street scene reconstruction method provided in the embodiment of the present application utilizes a set of unseen perspective images that have undergone image completion processing when constructing the second data set, thereby enriching the training data used to train the first street scene model and significantly improving the geometric reconstruction accuracy of unseen perspectives.

[0067] In an exemplary embodiment, the second data set is constructed by:

[0068] Firstly, image enhancement processing is performed on the partially occluded images in the obtained unseen view image set to obtain a first image set;

[0069] Secondly, image enhancement processing is performed on images in the captured training image set that have non-Lambertian illumination areas and areas lacking texture features, to obtain a second image set;

[0070] Finally, the second data set is constructed based on the first image set and the second image set.

[0071] Among them, the Lambert illumination model is one of the most basic illumination models, which assumes that the surface of an object is an ideal diffuse reflection surface and that light is reflected evenly in all directions. However, in the real world, many surfaces do not conform to the assumptions of the Lambert illumination model, and these areas are called non-Lambert illumination areas. In practical applications, non-Lambert illumination areas are often formed due to changing lighting conditions, such as strong shadows, specular reflections, highlights, and low-light areas. These areas cause distortion of image features, which increases the difficulty of feature matching and model reconstruction.

[0072] Texture feature-lacking areas refer to areas such as the sky, flat walls or roads that lack obvious texture features. These areas make feature point detection and matching difficult, thus affecting the integrity of reconstruction.

[0073] The street scene reconstruction method provided in the embodiment of the present application can perform image enhancement processing on images in the captured training image set that have non-Lambert lighting areas and areas lacking texture features. The captured image set may contain one or more images with non-Lambert lighting areas, and may contain one or more images with areas lacking texture features. Any image may contain only non-Lambert lighting areas, or may contain only areas lacking texture features, or may contain both non-Lambert lighting areas and areas lacking texture features.

[0074] The street scene reconstruction method provided in the embodiment of the present application utilizes a set of images from unseen perspectives that have undergone image completion processing, and also utilizes a set of images that have been captured and image enhanced when constructing the second data set, thereby further enriching the training data used to train the first street scene model, significantly improving the geometric reconstruction accuracy of unseen perspectives, performing image enhancement processing on non-Lambert lighting areas, thereby promoting the geometric consistency of the results, and performing image enhancement processing on areas lacking texture features, thereby reducing the difficulty of feature point detection and matching.

[0075] In an exemplary embodiment, when the second data set is constructed based on the first image set and the captured training image set, performing image enhancement processing on the partially occluded images in the obtained unseen view image set to obtain the first image set includes:

[0076] First, each image in the unseen viewpoint image set is detected to see if there is an artifact region, and the artifact region therein is masked;

[0077] Secondly, image completion processing is performed on the masked images, and a set of unseen viewpoint images in which the images have undergone image completion processing is used as the first image set.

[0078] Exemplarily, in the process of rendering the unseen view images, artifacts may be introduced due to view extrapolation and model uncertainty. Therefore, it is detected whether there is an artifact area in each image in the unseen view image set, and then the area with artifacts is masked, and then image completion processing is performed to obtain the unseen view image set after image completion processing and use it in subsequent iterative training.

[0079] Exemplarily, artifact detection can be achieved through artifact detection algorithms, including: pixel-level content analysis algorithms, which detect artifacts by analyzing the pixel-level features of the image. It can determine the location of image modification and is applicable to a variety of image processing scenarios; image format detailed analysis algorithms, which analyze the image format to determine whether the image has been modified after leaving the camera, and are suitable for detecting whether the image has been post-processed; neural network technology, which uses deep learning technology to identify and detect artifacts in images by training models. This method performs well in forged image detection and can effectively identify and classify artifacts.

[0080] In an exemplary embodiment, when the second data set is constructed based on the first image set and the second image set, performing image enhancement processing on the partially occluded images in the obtained unseen view image set to obtain the first image set includes:

[0081] First, each image in the unseen viewpoint image set is detected to see if there is an artifact region, and the artifact region therein is masked;

[0082] Secondly, performing image completion processing on the masked images, and taking a set of unseen viewpoint images in which the images have been subjected to the image completion processing as the first image set;

[0083] The step of performing image enhancement processing on images in the captured training image set having non-Lambertian illumination areas and areas lacking texture features to obtain a second image set comprises:

[0084] First, each image in the captured image set is detected to see if there is an artifact area, and the artifact area therein is masked;

[0085] Secondly, image completion processing is performed on the masked image, and a set of test images obtained by shooting and in which the images have been subjected to the image completion processing is used as the second image set.

[0086] For example, in the process of taking images at different viewing angles using a camera, artifacts may be introduced due to the existence of non-Lambertian lighting areas or areas lacking texture features. Therefore, it is detected whether there is an artifact area in each image in the captured image set, and then the area with artifacts is masked, and then image completion processing is performed to obtain a captured image set in which the images have been completed.

[0087] In an exemplary embodiment, if Figure 2 As shown, the image completion processing of the masked image includes:

[0088] Step 200, calculating the degree of mask uncertainty of each masked image;

[0089] Step 210: Use a two-dimensional diffusion model to perform image completion processing on the mask-processed image whose mask uncertainty degree is greater than a first preset threshold.

[0090] The mask is used to identify specific areas in the image, such as the region of interest or the background area. In some cases, there may be uncertainty in the boundaries or regions of the mask, which may come from image noise, segmentation errors, sensor accuracy, etc. The mask uncertainty degree is used to quantify the fuzziness or uncertainty of the mask boundaries or regions.

[0091] 2D Diffusion Models are a type of generative model that is mainly used for image generation tasks. Compared with traditional generative adversarial networks and variational autoencoders, 2D Diffusion Models learn the generation mechanism of images by simulating the process of data gradually diffusing from simple distributions (such as Gaussian noise) to complex data distributions.

[0092] In an exemplary embodiment, the first data set includes: a COLMAP data set, and the second data set includes: a COLMAP data set;

[0093] The three-dimensional model includes: a three-dimensional model represented by a 3D-GS method;

[0094] The two-dimensional diffusion model includes: a two-dimensional diffusion model trained by a denoising diffusion probabilistic model (Denoising Diffusion Probabilistic Models, DDPM) sampling method, or a two-dimensional diffusion model trained by a denoising diffusion implicit model (Denoising Diffusion Implicit Models, DDIM) sampling method.

[0095] Among them, the COLMAP dataset is a dataset designed and optimized for the COLMAP software and its related 3D reconstruction algorithms.

[0096] DDPM is a diffusion model sampling process based on the Markov hypothesis. The two-dimensional diffusion model trained in this sampling method can generate high-quality image, audio, text and other data. Its core idea is to learn the data generation mechanism by simulating the process of data gradually diffusing from noise to complex data.

[0097] In an exemplary embodiment, if Figure 3 As shown, the first data set is obtained by:

[0098] Step 300: performing image preprocessing on the images in the training image set;

[0099] Step 310: extract feature points from each image that has undergone image preprocessing, and perform feature point matching between different images that have undergone image preprocessing based on the extracted feature points and using a bidirectional nearest neighbor matching method to obtain multiple pairs of matching images that have feature point matching pairs;

[0100] Step 320: Estimate the initial pose of the camera using the obtained multiple pairs of matching images;

[0101] Step 330: Perform sparse reconstruction based on the obtained initial pose and feature point matching pairs to obtain other poses of the camera other than the initial pose and a three-dimensional point cloud for reconstructing the street scene;

[0102] Step 340: performing reprojection error minimization processing on all camera poses and the obtained three-dimensional point cloud bundles;

[0103] Step 350: construct the first data set according to the camera's intrinsic parameters, the processed posture and the processed three-dimensional point cloud, and the matching information of multiple pairs of matching images.

[0104] In the field of computer vision and image processing, feature point matching is a basic step for many tasks, such as image stitching, 3D reconstruction, and target tracking. Bidirectional Nearest Neighbor (BNN) is one of the commonly used feature point matching strategies, which can improve the accuracy and robustness of feature matching.

[0105] Exemplarily, feature points of each image that has undergone image preprocessing can be extracted by a feature point extraction algorithm. Feature point extraction algorithms include: Scale-Invariant Feature Transform (SIFT) algorithm, Speeded-Up Robust Features (SURF) algorithm, Features from Accelerated Segment Test (FAST) algorithm, and Harris corner detection algorithm. Among them, the SIFT algorithm is widely used in image matching, object recognition, image stitching, three-dimensional reconstruction and other fields. Its core idea is to detect stable feature points in different scale spaces of the image and generate feature descriptors with scale invariance and rotation invariance. The SIFT algorithm realizes feature detection and description through scale space construction, key point detection, key point direction assignment, and feature descriptor generation steps.

[0106] In an exemplary embodiment, the captured training image set includes: an image sequence in which the overlapping area between adjacent images is not less than a second preset threshold.

[0107] For example, the presence of a certain overlap area between adjacent images can ensure the smooth progress of the subsequent feature matching and posture estimation process. In practical applications, the second preset threshold value can be 60%.

[0108] The embodiment of the present application also provides a method for reconstructing a street scene, which performs street reconstruction based on the captured RGB picture, such as Figure 4 As shown, the following steps are included:

[0109] Step 400: construct a COLMAP dataset based on the RGB image sequence.

[0110] The RGB picture sequence in this step corresponds to the RGB picture set obtained by shooting in the above embodiment. And this step may include the following sub-steps:

[0111] Step 400a: image acquisition and preprocessing.

[0112] Use a high-resolution RGB camera to capture the target street scene with a certain coverage and overlap rate, ensuring that there is at least 60% overlap between adjacent images in the image sequence for subsequent feature matching and pose estimation. The collected RGB images need to be preprocessed, including denoising, color correction, and geometric correction, to improve image quality and feature detection accuracy.

[0113] Step 400b: feature extraction and matching.

[0114] The SIFT algorithm is used to extract feature points from each preprocessed RGB image, and the feature points extracted from each image are represented as a set of descriptor vectors:

[0115] D i ={d 1 ,d 2 ,…,d n}

[0116] Among them, D i represents the descriptor set of the i-th image, d p is the descriptor of the pth feature point, p = 1, 2…n.

[0117] The nearest neighbor search algorithm is used to perform feature matching between image pairs, and a two-way verification strategy, namely, the two-way nearest neighbor, is used to eliminate false matches and generate a set of matching pairs:

[0118]

[0119] in, represents the set of feature point matching pairs between the i-th image and the j-th image, represents the descriptor of the kth feature point in the i-th image, Represents the descriptor of the lth feature point in the jth image.

[0120] Step 400c: initial pose estimation.

[0121] For each pair of matching images, the fundamental matrix F is estimated using the random sampling consensus algorithm (RANSAC) to remove outliers and retain inlier matches:

[0122] Fx i × j =0

[0123] Among them, xi and x j Respectively represent the homogeneous coordinates of the matching feature points in the i-th image and the j-th image.

[0124] Select the image pair with the most matching pairs of feature points as the initial view, and use the basic matrix F to decompose the initial relative pose of the camera (hereinafter referred to as the initial pose):

[0125] F=K ―T [t] × R ―1

[0126] Where R is the rotation matrix of the camera, t is the translation vector of the camera, K is the camera intrinsic parameter matrix, [t] × is the antisymmetric matrix of the translation vector t.

[0127] Step 400d: sparse reconstruction.

[0128] Based on the initial pose and feature point matching pairs, the position of the 3D point is calculated by triangulation:

[0129] X=Triangulate(P i ,P j ,x i ,x j )

[0130] Among them, P i and P j are the projection matrices of the i-th and j-th images respectively, and X is the homogeneous coordinates of the three-dimensional point.

[0131] Add the other RGB images in the sequence to the reconstruction process in turn to estimate the camera pose and add 3D points. Each time a new image is added, perform the following steps:

[0132] (1) Solving the PnP problem: Use the EPnP algorithm (Efficient Perspective-n-Point) to estimate the camera pose of a new image from known 3D points and corresponding 2D image points.

[0133] (2) Local triangulation: Triangulation is performed to add new 3D points based on matching pairs of feature points between the new image and the reconstructed view.

[0134] Step 400e: global optimization.

[0135] Bundle all camera poses and 3D points to minimize the reprojection error:

[0136] ρ is a robust kernel function used to reduce the impact of outliers, R iis the rotation matrix of the i-th image, t i is the translation vector of the i-th image, X q are the homogeneous coordinates of the qth three-dimensional point.

[0137] Step 400f, constructing a data set.

[0138] The camera internal parameters (including focal length, principal point, distortion coefficient and other parameters), the reconstructed camera pose (also called camera external parameters), 3D point cloud and image matching information are stored in the data format supported by COLMAP to generate a complete COLMAP data set structure. The accuracy of the reconstruction results is checked through visualization tools to ensure the rationality and consistency of the camera pose and 3D point cloud.

[0139] Step 410: Train a three-dimensional model represented by the 3D-GS method according to the COLMAP dataset.

[0140] This step may include the following sub-steps:

[0141] Step 410a: Data preparation.

[0142] Using the COLMAP dataset constructed in step S1, the camera pose and 3D point cloud information are extracted and used as the input of the model.

[0143] Step 410b: model training.

[0144] Using the 3D-GS algorithm, the street scene is represented as a set of Gaussian primitives:

[0145]

[0146] Among them, each Gaussian primitive g i It includes parameters such as position, scale, color, etc. By optimizing these parameters, the model can accurately reconstruct the radiation field and geometric structure of the street scene under the training perspective.

[0147] The loss function Defined as the pixel error between the reconstructed image and the original image:

[0148] The kth RGB image obtained.

[0150] Through the back propagation algorithm, the parameters of the Gaussian primitive are iteratively optimized until the loss function converges, and the trained initial model (which can be recorded as model M1) is obtained.

[0151] Step 420: Obtain an unseen viewing angle RGB image sequence according to the initial model.

[0152] Using model M1, generate RGB image sequences from unseen perspectives. The specific steps are as follows:

[0153] I unseen = Render(M1,P unseen )

[0154] Among them, P unseen Represents the camera projection matrix of the unseen perspective. The Render function generates the corresponding RGB image based on the model and the camera projection matrix.

[0155] Step 430: Detect the artifact area of ​​the unseen viewing angle RGB picture sequence and perform masking.

[0156] This step may include the following sub-steps:

[0157] Step 430a: Artifact detection.

[0158] In the rendered RGB image of an unseen view, artifacts may be introduced due to view extrapolation and model uncertainty. Advanced artifact detection algorithms are used to identify artifact areas in the image:

[0159]

[0160] in, A mask image representing the detected artifact area, where the masked area is marked as 1 and the rest of the area is marked as 0.

[0161] Step 430b: mask processing.

[0162] Perform mask processing on the detected artifact area to generate a mask image:

[0163]

[0164] Among them, ⊙ represents the pixel-by-pixel multiplication operation, I masked is the image after masking.

[0165] Step 440: Use a two-dimensional diffusion model to complete the image.

[0166] This step may include the following sub-steps:

[0167] Step 440a: Select a diffusion model.

[0168] Select a two-dimensional diffusion model suitable for the image completion task, such as a two-dimensional diffusion model trained using DDPM sampling.

[0169] Step 440b: image completion.

[0170] Calculating the degree of mask uncertainty Among them, H is the height of the RGB image, W is the width of the RGB image, It is the mask area mark of the pixel in the mth row and nth column on the RGB image.

[0171] If Ω>0.1, the masked image is input into the two-dimensional diffusion model for high-quality image completion to generate the completed image I inpainted :

[0172]

[0173] Among them, the DiffusionInpaint function is based on the mask area Generate context-consistent completion content and obtain artifact-free completion image I inpainted .

[0174] Step 450: Add the completed image to the original image data to obtain a new COLMAP data set.

[0175] This step may include the following sub-steps:

[0176] Step 450a: Update the data set.

[0177] The completed unseen perspective RGB image I inpainted Join the original RGB image sequence to form a new image dataset:

[0178]

[0179] Step 450b: construct a new COLMAP dataset.

[0180] Using the method described in step 400, the updated image dataset I new The new COLMAP dataset is generated by re-performing feature extraction, matching, pose estimation and sparse reconstruction. This dataset contains information of the original image and the completed unseen view images, which enhances the coverage and geometric information of the scene.

[0181] Step 460: Continue training M1 according to the new COLMAP dataset to obtain an updated model.

[0182] The updated model can be recorded as M2. This step can include the following sub-steps:

[0183] Step 460a: Data preparation.

[0184] The new COLMAP dataset generated in step 450 is used to extract the camera pose and 3D point cloud information, and these data are used as input for further training the model M1.

[0185] Step 460b: model training.

[0186] The same 3D-GS algorithm as in step 410 is used for training using the new data set:

[0187]

[0188] By optimizing the parameters of a new set of Gaussian primitives, the optimized model (which can be recorded as model M2) can more accurately reconstruct the scene, including areas where artifacts and geometric distortions previously existed.

[0189] Loss Function It is also defined as the pixel error between the reconstructed image and the original image, but this training includes the completed image, which improves the model's performance in unseen perspectives and complex areas:

[0190]

[0191] Through the back propagation algorithm, the parameters of the Gaussian primitive are iteratively optimized until the loss function converges to obtain the optimized model M2.

[0192] The street scene reconstruction method provided in the embodiment of the present application utilizes a set of unseen perspective RGB images that have undergone image completion processing when constructing a new data set, thereby enriching the training data used to train the street scene model, realizing iterative optimization of the model, and significantly improving the geometric reconstruction accuracy of unseen perspectives.

[0193] The above embodiments of the present application solve the problem of the existence of occluded areas, non-Lambertian illumination areas, and areas lacking texture features. However, in order to solve the problem of the existence of occluded areas, non-Lambertian illumination areas, and areas lacking texture features, there are many solutions for scene reconstruction using RGB images in the related art, which are mainly divided into the following two categories:

[0194] 1. Structure reconstruction method based on visual feature matching

[0195] This type of method recovers the three-dimensional structure by detecting and matching explicit feature points in the image. Typical representatives include the Structure from Motion (SfM) method and the Multi-view Stereo (MVS) algorithm.

[0196] COLMAP: A widely used SfM and MVS tool that uses robust feature detection and matching algorithms to reconstruct sparse and dense point clouds of a scene from multiple images. However, it has limited performance in dealing with occlusions, illumination changes, and weakly textured areas, which can easily lead to incomplete or distorted reconstructions.

[0197] DUSt3R series: These methods improve on traditional SfM algorithms by using data-driven models to match feature points between image pairs.

[0198] However, methods based on visual feature matching rely on explicit feature points in the image. For areas with weak texture or repeated texture, feature points are difficult to detect or easily confused, resulting in matching errors.

[0199] 2. Implicit Representation Method Based on New Perspective Synthesis

[0200] This type of method uses machine learning techniques such as neural networks to learn implicit representations of scenes to achieve synthesis of new perspectives and 3D reconstruction. It mainly includes the following algorithms:

[0201] NeRF (Neural Radiance Fields): NeRF uses a neural network to learn the volume rendering function of the scene to achieve high-quality new perspective synthesis. It can generate images with rich details, but the training process is time-consuming.

[0202] 3D-GS: This method uses Gaussian distribution to implicitly model the scene, which improves rendering efficiency and quality. However, its performance still needs to be improved when dealing with complex lighting conditions and occlusion problems.

[0203] ViewCrafter: It attempts to solve the problem of NeRF application in large-scale scenes. It ensures the integrity of the point cloud from all perspectives by completing the missing parts in the point cloud.

[0204] However, implicit representation methods based on new view synthesis have limited ability to handle lighting changes and are unable to cope with complex situations in actual street scenes.

[0205] In summary, when dealing with complex street scenes, existing scene reconstruction methods based on RGB images are subject to problems such as occlusion, illumination changes, and weak textures, making it difficult to obtain complete and high-precision reconstruction results.

[0206] In addition, existing methods also have the following problems of high computational complexity or storage requirements. Among them, the computational complexity or storage requirements of the structure reconstruction method based on visual feature matching are high, which limits its application in large-scale scenes. For example, when dealing with large-scale scenes, the computational and storage requirements of the DUSt3R series of algorithms are greatly increased compared to traditional feature engineering. Although the implicit representation method performs well in synthesizing new perspectives, the demand for computing and storage resources becomes the main bottleneck in large-scale scenarios. For example, as the number of new perspectives increases, the demand for video memory and storage space in the NeRF method increases sharply, thus limiting its practicality.

[0207] The street scene reconstruction method provided in the embodiment of the present application can solve the problem of high computational complexity or storage requirement. The following is an example of comparison with the most advanced MASt3R in the related technology. The intermediate files of the street scene reconstruction method provided in the embodiment of the present application are mainly generated by COLMAP. Therefore, by comparing the intermediate file occupancy of MASt3R and COLMAP, the computational complexity or storage requirement of the street scene reconstruction method provided in the embodiment of the present application and MASt3R can be compared.

[0208] Assume that the input data is 100 images, and the resolution of each image is 1960×1024.

[0209] MASt3R needs to go through the steps of image loading, image pair generation, sparse global matching, optimization phase, and 3D scene generation. The estimated size of the intermediate files is as follows: forward reasoning takes up 34.65GB, the optimization phase relies on cache files and does not generate new files, and 3D scene generation takes up 150MB, for a total of approximately 34.8GB of storage space.

[0210] In the street scene reconstruction method provided in the embodiment of the present application, the intermediate file COLMAP generates file sizes of 208MB, 40MB, 5MB, and 3.7GB in the feature extraction stage, feature matching stage, sparse reconstruction stage, and dense reconstruction stage, respectively, occupying a total of 4GB of storage space.

[0211] It can be seen that the street scene reconstruction method provided in the application embodiment can greatly reduce the storage space occupied. The reason why the street scene reconstruction method provided in the application embodiment occupies less storage space than the method MASt3R-SfM that uses MASt3R to complete the scene reconstruction work is mainly due to the following aspects:

[0212] 1. Feature extraction and matching method:

[0213] MASt3R-SfM uses deep learning to extract high-dimensional features and store dense matching relationships, and the file is much larger than COLMAP's traditional SIFT features and sparse matching data.

[0214] 2. Cache strategy:

[0215] MASt3R-SfM caches dense point clouds and matching data for each pair of images, resulting in larger file sizes, while COLMAP uses a compact database to store keypoints and matching point indices, resulting in smaller file sizes.

[0216] 3. Triangulation strategy:

[0217] The sparse point cloud model file of COLMAP is very small, while MASt3R-SfM generates dense point clouds during the matching stage, and the file size expands rapidly.

[0218] 4. Additional overhead of deep learning models:

[0219] Deep learning feature extraction and matching context information requires larger storage space.

[0220] The street scene reconstruction method provided in the embodiment of the present application can be used for the street scene reconstruction process as follows: Figure 5 As shown in the figure, after the RGB image sequence undergoes initial geometric estimation such as SfM, a calibrated RGB image sequence and camera pose are obtained, and then after 2D supervision and training, a 3D model represented by the 3D-GS method is used to obtain an unseen perspective rendered image sequence, and then uncertainty estimation such as DINO is performed to obtain an artifact mask sequence, and finally the image is completed based on a 2D diffusion model to obtain an optimized rendered image sequence. The dotted line in the figure indicates that the execution is executed when the uncertain area ratio Ω>0.1.

[0221] The present application also provides a street scene reconstruction device, such as Figure 6 As shown, including:

[0222] An acquisition unit 500 is used to acquire a target image set shot for a target street scene;

[0223] A processing unit 510 is used to reconstruct a target street scene based on the obtained target image set and the target street scene model;

[0224] The target street scene model is pre-trained based on a first data set and a second data set. The first data set is constructed based on a training image set shot for the target street scene and is used to train a pre-established three-dimensional model to obtain an initial street scene model. The second data set is constructed based on an unseen perspective image set and a shot training image set. The unseen perspective image set is obtained using the obtained initial street scene model.

[0225] The street scene reconstruction device provided in the embodiment of the present application obtains a target image set shot for a target street scene; reconstructs the target street scene based on the obtained target image set and a target street scene model; wherein the target street scene model is pre-trained based on a first data set and a second data set, the first data set is constructed based on a training image set shot for the target street scene, and is used to train a pre-established three-dimensional model to obtain an initial street scene model; the second data set is constructed based on an unseen perspective image set and a shot training image set, and the unseen perspective image set is obtained using the obtained initial street scene model. Since the target street scene model used is obtained through iterative optimization of the initial street scene model, the geometric reconstruction accuracy of the street scene is improved when it is used for street scene reconstruction.

[0226] The street scene reconstruction device provided in this embodiment belongs to the same application concept as the street scene reconstruction method provided in the above embodiment of this application, and can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the street scene reconstruction method. For technical details not fully described in this embodiment, please refer to the specific processing content of the street scene reconstruction method provided in the above embodiment of this application, and will not be repeated here.

[0227] The functions implemented by the above acquisition unit 500 and processing unit 510 may be implemented by the same or different processors, respectively, and the embodiments of the present application are not limited thereto.

[0228] It should be understood that each unit in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units above are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all units above. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in the form of a processor calling software, and the remaining part can be implemented in the form of a hardware circuit.

[0229] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0230] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0231] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0232] The present application also provides an electronic device, such as Figure 7 As shown, it includes: a memory 600 and a processor 610;

[0233] The memory 600 is connected to the processor 610 and is used to store programs;

[0234] The processor 610 is used to implement the street scene reconstruction method described in any of the above embodiments by running the program in the memory 600.

[0235] Specifically, the electronic device may further include: a bus, a communication interface 620 , an input device 630 and an output device 640 .

[0236] The processor 610, the memory 600, the communication interface 620, the input device 630 and the output device 640 are connected to each other via a bus.

[0237] A bus may include a pathway that transfers information between components of a computer system.

[0238] The processor 610 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0239] The processor 610 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0240] The memory 600 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 600 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0241] The input device 630 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0242] Output device 640 may include a device that allows information to be output to a user, such as a display screen, printer, speaker, etc.

[0243] The communication interface 620 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0244] The processor 610 executes the program stored in the memory 600 and calls other devices, which can be used to implement each step of any street scene reconstruction method provided in the above embodiments of the present application.

[0245] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for reconstructing street scenes according to various embodiments of the present application described in any of the above embodiments of this specification.

[0246] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0247] In addition, an embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for reconstructing a street scene described in any of the above embodiments is implemented.

[0248] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0249] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0250] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0251] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0252] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0253] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0254] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0255] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0256] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0257] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0258] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for reconstructing a street scene, characterized in that: include: Acquire a target image set shot for a target street scene; Reconstructing the target street scene based on the obtained target image set and the target street scene model; The target street scene model is pre-trained based on a first data set and a second data set. The first data set is constructed based on a training image set shot for the target street scene and is used to train a pre-established three-dimensional model to obtain an initial street scene model. The second data set is constructed based on an unseen perspective image set and a shot training image set. The unseen perspective image set is obtained using the obtained initial street scene model.

2. The reconstruction method according to claim 1, characterized in that: The second dataset is constructed in the following way: Performing image enhancement processing on partially occluded images in the obtained unseen view image set to obtain a first image set; constructing the second data set based on the first image set and the captured training image set; or, Perform image enhancement processing on partially occluded images in the obtained unseen view image set to obtain a first image set; perform image enhancement processing on images in non-Lambert lighting areas and areas lacking texture features in the captured training image set to obtain a second image set; and construct the second data set based on the first image set and the second image set.

3. The reconstruction method according to claim 2, characterized in that: In the case where the second data set is constructed based on the first image set and the captured training image set, the image enhancement process is performed on the partially occluded images in the obtained unseen view image set to obtain the first image set, including: Detecting whether each image in the set of unseen viewing angle images has an artifact region, and masking the artifact region therein; The masked images are subjected to image completion processing, and a set of unseen viewpoint images in which the images have undergone image completion processing is used as the first image set.

4. The reconstruction method according to claim 2, characterized in that: In the case where the second data set is constructed based on the first image set and the second image set, performing image enhancement processing on the partially occluded images in the obtained unseen view image set to obtain the first image set includes: Detecting whether each image in the set of unseen viewing angle images has an artifact region, and masking the artifact region therein; Performing image completion processing on the masked images, and taking a set of unseen viewpoint images in which the images have been subjected to the image completion processing as the first image set; The step of performing image enhancement processing on images in the captured training image set having non-Lambertian illumination areas and areas lacking texture features to obtain a second image set comprises: Detecting whether there is an artifact region in each image in the captured image set, and masking the artifact region therein; The masked image is subjected to image completion processing, and a set of test images obtained by shooting and in which the image is subjected to image completion processing is used as the second image set.

5. The reconstruction method according to claim 3 or 4, characterized in that: The image completion processing of the masked image includes: Calculate the degree of mask uncertainty for each masked image; The two-dimensional diffusion model is used to perform image completion processing on the mask-processed image whose mask uncertainty degree is greater than a first preset threshold.

6. The reconstruction method according to claim 5, characterized in that: The first data set includes: a COLMAP data set, and the second data set includes: a COLMAP data set; The three-dimensional model includes: a three-dimensional model represented by a 3D-GS method; The two-dimensional diffusion model includes: a two-dimensional diffusion model trained in a DDPM sampling manner.

7. The reconstruction method according to claim 1, characterized in that: The first data set is obtained by: Performing image preprocessing on the images in the training image set; Extracting feature points from each image that has undergone image preprocessing, performing feature point matching between different images that have undergone image preprocessing based on the extracted feature points and using a bidirectional nearest neighbor matching method, and obtaining multiple pairs of matching images with feature point matching pairs; Estimate the initial pose of the camera using the obtained multiple pairs of matching images; Sparse reconstruction is performed based on the obtained initial pose and feature point matching pairs to obtain other poses other than the initial pose of the camera and a three-dimensional point cloud for reconstructing the street scene; Minimize the reprojection error for all camera poses and the obtained 3D point cloud bundles; The first data set is constructed according to the camera's intrinsic parameters, the processed posture and the processed three-dimensional point cloud, and the matching information of multiple pairs of matching images.

8. The reconstruction method according to claim 1, characterized in that: The captured training image set includes: an image sequence in which the overlapping area between adjacent images is not less than a second preset threshold.

9. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the street scene reconstruction method as described in any one of claims 1 to 8 by running the program in the memory.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for reconstructing a street scene as claimed in any one of claims 1 to 8 is implemented.

Images