Image processing method and device and storage medium

Abstract

The invention discloses an image processing method and device and a storage medium, and the method comprises the steps: obtaining an input image sequence of a target object, and determining a mask image of the target object based on an image frame of the input image sequence; selecting a key frame from the input image sequence, and determining camera pose information of a visual angle corresponding to the key frame; and based on the key frame, the mask image and the camera pose information, performing three-dimensional model reconstruction on the target object. The technical problem that when an existing three-dimensional reconstruction technology based on a monocular camera reconstructs an object in a real scene, corresponding reconstruction benefit requirements are difficult to meet is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] Currently, computer 3D vision is increasingly used in autonomous driving, virtual reality / augmented reality, artificial intelligence robots, etc., and 3D reconstruction, as one of the core tasks of computer 3D vision, has received increasing attention in recent years.

[0003] Compared with 3D reconstruction algorithms that rely on sensors, 3D reconstruction technologies based on monocular cameras, such as 3D reconstruction methods based on implicit representation, show many advantages in terms of cost, operation, and structure. However, most of the existing 3D reconstruction methods based on implicit representation are trained using pre-calibrated mask images in the dataset, which cannot accurately reconstruct objects in real scenes and require a large amount of manual data preprocessing processes, making it difficult to meet the current efficiency requirements of 3D reconstruction based on monocular cameras. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an image processing method, device and storage medium, which aim to solve the technical problem that the existing three-dimensional reconstruction technology based on a monocular camera is difficult to meet the corresponding reconstruction efficiency requirements when reconstructing objects in a real scene.

[0005] The present application provides an image processing method, the method comprising:

[0006] Acquire an input image sequence of a target object, and determine a mask image of the target object based on image frames of the input image sequence;

[0007] Selecting a key frame from the input image sequence, and determining the camera pose information of the key frame corresponding to the viewing angle;

[0008] Based on the key frame, the mask image and the camera pose information, a three-dimensional model of the target object is reconstructed.

[0009] The present application also provides an image processing device, the device comprising:

[0010] A mask generation module, used for acquiring an input image sequence of a target object, and determining a mask image of the target object based on image frames of the input image sequence;

[0011] A key frame selection and posture calibration module, used to select a key frame from the input image sequence and determine the camera posture information of the key frame corresponding to the viewing angle;

[0012] A three-dimensional reconstruction module is used to reconstruct a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information.

[0013] The present application also provides an image processing device, which is a physical node device, and includes: a memory, a processor, and a program of the image processing method stored in the memory and executable on the processor. When the program of the image processing method is executed by the processor, the steps of the image processing method described above can be implemented.

[0014] In order to achieve the above-mentioned purpose, a storage medium is further provided, on which an image processing program is stored, and when the image processing program is executed by a processor, the steps of any of the above-mentioned image processing methods are implemented.

[0015] The present application provides an image processing method, device and storage medium. Compared with the related art in which the three-dimensional reconstruction technology based on a monocular camera reconstructs objects in a real scene, it is difficult to meet the corresponding reconstruction efficiency requirements. In the present application, an input image sequence of a target object is obtained, and a mask image of the target object is determined based on the image frame of the input image sequence; a key frame is selected from the input image sequence, and the camera pose information of the key frame corresponding to the viewing angle is determined; based on the key frame, the mask image and the camera pose information, a three-dimensional model of the target object is reconstructed. In the present application, by generating a mask image of the target object and extracting important information of the target object through key frame selection (including the camera pose information of the corresponding viewing angle), the three-dimensional model of the target object is reconstructed. It can be understood that the above process does not require pre-calibration of mask images for training, avoids the need for a large number of manual data preprocessing processes, reduces noise and unnecessary calculations in the reconstruction process, and because the present application does not need to introduce new sensors, important information of the target object can be extracted, and the reconstruction of the three-dimensional model is realized, which saves costs, that is, in this embodiment, the reconstruction efficiency of the corresponding target object is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the flow chart of the first embodiment of the image processing method of the present application;

[0017] Figure 2 This is a detailed flowchart of step S20 in the first embodiment of the image processing method of the present application;

[0018] Figure 3 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

[0019] Figure 4A schematic diagram of the structure of an image processing device involved in the image processing method of the present application;

[0020] Figure 5 Schematic diagram of the process flow of the image processing method involved in the image processing method of this application. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] The present application embodiment provides an image processing method. In one embodiment of the image processing method of the present application, the image processing method is applied to an image processing device. Figure 1 , the method comprising:

[0023] Step S10, acquiring an input image sequence of a target object, and determining a mask image of the target object based on image frames of the input image sequence;

[0024] Step S20, selecting a key frame from the input image sequence, and determining the camera pose information of the key frame corresponding to the viewing angle;

[0025] Step S30: reconstructing a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information.

[0026] In this embodiment, basic terms are first described:

[0027] SDF (Signed Distance Function): Signed distance function, which represents the distance and direction information from the surface of an object to a certain point;

[0028] BSDF (Bidirectional Scattering Distribution Function): Bidirectional scattering distribution function, used to describe the scattering behavior of a material or object to incident light;

[0029] MC (Marching Cubes): A surface reconstruction algorithm for 3D voxel data that converts discrete voxel data into a smooth 3D mesh model;

[0030] DMTet (Dual Marching Tetrahedra): An improved version of the Marching Tetrahedra algorithm;

[0031] NeRF (Neural Radiance Fields): A neural network-based method for reconstruction and rendering of 3D scenes;

[0032] MLP (Multi layer Perceptron): an artificial neural network model;

[0033] SG (Spherical Gaussian function): spherical Gaussian function;

[0034] IOU (Intersection over Union): a commonly used evaluation indicator in computer vision tasks. That is, the background involved in this embodiment is:

[0035] Currently, computer 3D vision is increasingly used in autonomous driving, virtual reality / augmented reality, artificial intelligence robots, etc., and 3D reconstruction, as one of the core tasks of computer 3D vision, has received increasing attention in recent years. Compared with the use of depth sensors, the images in image-based 3D reconstruction technology do not contain accurate depth information, cannot reflect 3D geometric structures, and lack uniform lighting conditions, making it difficult to directly generate high-precision 3D models.

[0036] Currently, there are 3D reconstruction algorithms that rely on sensors to reconstruct single targets. However, 3D reconstruction algorithms that rely on sensors are highly dependent on professional equipment and have high acquisition costs, which hinders large-scale use by users. In addition, the captured depth maps are either too sparse and lose a lot of details.

[0037] In comparison, the 3D reconstruction technology based on monocular camera presents many advantages in terms of cost, operation and structure. Therefore, image processing is usually performed based on monocular 2D RGB images.

[0038] However, the current 3D reconstruction technology based on monocular cameras has the following shortcomings and limitations in terms of universalization, which restricts its large-scale promotion and application, specifically:

[0039] First, traditional matching-based dense reconstruction algorithms (such as COLMAP) rely heavily on the correspondence of different visual features. For areas with a large number of no textures, weak textures, and repeated textures, such as repeated patterns, solid color surfaces, and reflective surfaces, holes will be generated during reconstruction, resulting in poor results. In addition, matching-based dense reconstruction algorithms have the problems of high computational complexity and long time consumption during dense reconstruction and subsequent Poisson reconstruction.

[0040] Second, deep neural network-based and data-driven (such as depth-based and TSDF-based) methods can effectively alleviate the textureless problem of the above methods by utilizing various geometric priors learned from a large amount of data. However, such methods extract image features through multi-layer convolutions, and the three-dimensional convolution layers they rely on have huge computational overhead, high computation time and runtime memory requirements, and cannot complete high-precision reconstruction.

[0041] Third, the reconstruction algorithm based on neural radiation field (NeRF) requires the collection of a large number (hundreds) of photos from different perspectives to achieve good results, which leads to low overall efficiency of the algorithm, slow training speed and long inference time.

[0042] Based on this, current research uses the technology of neural implicit surface reconstruction to achieve image processing. However, some methods learn implicit surfaces through volume rendering and reduce the sample area of ​​volume rendering to improve the reconstruction quality. However, the above methods need to extract geometric features from a single pixel along the light sampling direction, which may hinder the sharing of adjacent information between sampling points around the surface, and the quality of the reconstructed surface depends largely on the ability of the MLP network to induce spatial relationships between adjacent points. Some methods that directly reconstruct the three-dimensional model of an object from multi-view images take a lot of time and have low reconstruction accuracy due to the lack of mask image priors. Many methods based on mask supervision have only been experimented on datasets, and no effective mask generation technology has been provided, making it impossible to reconstruct high-quality images collected in real environments.

[0043] Based on this, an image processing method in the present application is proposed. In the method of the present application, a mask image of the target object is generated and important information of the target object is extracted through key frame selection (including camera pose information of the corresponding viewing angle), and then, reconstruction of a three-dimensional model of the target object is achieved. That is, the present application does not require pre-calibrated mask images for training, thus avoiding the need for a large amount of manual data preprocessing processes, reducing noise and unnecessary calculations in the reconstruction process, and since the present application does not require the introduction of new sensors to extract important information of the target object, and then, reconstruction of the three-dimensional model is achieved, this saves costs, that is, in this embodiment, the reconstruction efficiency of the corresponding target object is improved.

[0044] In the present application, a high-quality foreground object mask is generated by screening preset evaluation indicators and / or performing preset preprocessing operations, thereby providing accurate target object information for subsequent three-dimensional reconstruction and improving the quality of the reconstruction result.

[0045] In the present application, the intermediate mask is fused with the corresponding image frame to avoid missing important information, provide accurate target object information for subsequent three-dimensional reconstruction, and improve the quality of the reconstruction result.

[0046] In the present application, key frames are selected, and feature extraction, feature matching and camera pose solution are performed on the selected key frames to obtain the camera pose information under the corresponding viewing angle of the key frames. Therefore, the amount of processed data and unnecessary calculations can be reduced, thereby speeding up image processing and improving the real-time and efficiency of the system.

[0047] In the present application, based on the loss information, the network parameters of the single-target model of the preset neural implicit surface are updated to achieve accurate reconstruction of the target object.

[0048] In the present application, a rendered image at a corresponding viewing angle is accurately generated, thereby improving the quality of the reconstruction result.

[0049] The specific steps are as follows:

[0050] Step S10, acquiring an input image sequence of a target object, and determining a mask image of the target object based on image frames of the input image sequence;

[0051] As an example, the image processing method is applied to an image processing apparatus, which belongs to an image processing device, and the image processing device may specifically be a computer device.

[0052] As an example, the target object may be a single target object, and the following is a specific description taking the case where the target object is a single target object as an example.

[0053] In this application, the structure of the image processing device is as follows: Figure 4 As shown, it includes 1. mask generation; 2. key frame selection and pose calibration; 3. single target reconstruction.

[0054] As an example, the present application is applicable to a variety of computer vision tasks that need to be performed on a computer device, and mainly runs on Linux and Windows desktop or command line operating systems that can run Python.

[0055] As an example, when an image processing method is running, it has certain requirements on hardware, such as the computing power of a graphics card, and is usually deployed on an NVIDIA graphics card.

[0056] As an example, when an image processing method is running, higher graphics card computing power can speed up the training process, but computing power only affects the training time and will not affect the accuracy of the final inference results.

[0057] As an example, the target object specifically refers to a single target object, which may be an animal or a person, without any specific limitation.

[0058] As an example, an input image sequence of a target object is acquired in real time.

[0059] As an example, the input image sequence is an input image sequence acquired without drastic visual changes, such as changes in illumination intensity and changes in the position and posture of the target object;

[0060] As an example, the input image sequence may be a multi-view input image sequence;

[0061] As an example, collecting a large number of images of objects from different perspectives can help improve the accuracy of network reconstruction. It is sufficient that there is no severe shaking or low clarity when shooting.

[0062] As an example, the image frame is an RGB picture;

[0063] As an example, the input image sequence is specifically a multi-view RGB image;

[0064] As an example, the mask image of the target object is determined based on a certain image frame of the input image sequence.

[0065] As an example, a mask image of the target object is determined based on a plurality of image frames of the input image sequence.

[0066] As an example, the mask image specifically refers to a foreground object mask;

[0067] As an example, the foreground object mask is a mask of the target object after removing the background.

[0068] Reference Figure 2 The step of determining the mask image of the target object based on the image frames of the input image sequence comprises:

[0069] Step S11, assigning each pixel in the image frame to a different object instance through a preset instance segmentation network structure;

[0070] In this embodiment, the mask generation module implements the process of determining the mask image of the target object based on the image frames of the input image sequence.

[0071] As an example, the mask generation module may be composed of an instance segmentation network submodule and a mask preprocessing submodule.

[0072] As an example, the ffmpeg tool is used to decode the input video frame by frame and convert it into image frames, and each image frame is used as the input of a preset instance segmentation network structure to generate mask layers for each instance in the image.

[0073] As an example, the preset instance segmentation network structure (instance segmentation network submodule) can be a pre-trained convolution-based YOLACT network structure;

[0074] As an example, the instance segmentation network submodule uses a pre-trained convolution-based YOLACT network to process image frames in the input image sequence;

[0075] As an example, the preset instance segmentation network structure is a trained model that can distinguish the target object from the background in the image (through the preset instance segmentation network structure, each pixel in the image frame is assigned to a different object instance, and the foreground and background are distinguished accordingly). Then, it processes the image frames in the input image sequence to distinguish the target object from the corresponding background.

[0076] Step S12, determining the target object from the object instance, distinguishing the target object and the background in the image frame, and obtaining an initial mask of the target object;

[0077] As an example, a preset instance segmentation network structure determines the probability that each pixel belongs to a target object, thereby distinguishing the target object and the background in the image frame to obtain an initial mask of the target object.

[0078] Step S13, performing a screening operation on the initial mask based on preset evaluation indicators to obtain a foreground object mask corresponding to the target object, wherein the preset evaluation indicators include the intersection-over-union ratio of the target object in the image frame, the screen ratio, and the distance from the local center area.

[0079] As an example, the mask preprocessing submodule may filter the corresponding initial mask by a foreground evaluation index (preset evaluation index) to obtain a foreground object mask corresponding to the target object.

[0080] As an example, the preset evaluation indicators include the intersection-over-union (IOU) ratio of the target object in the image frame, the screen ratio, and the distance from the local center area.

[0081] As an example, the intersection-over-union ratio of each target mask and the central 50% area of ​​the image is first calculated, and the target mask with the largest intersection-over-union ratio is selected as the reconstructed target mask, and the mask with a screen ratio greater than 15% is used as the foreground object mask.

[0082] The step of performing a screening operation on the initial mask based on a preset evaluation index to obtain a foreground object mask corresponding to the target object includes:

[0083] Performing a screening operation on the initial mask based on a preset evaluation index;

[0084] A preset preprocessing operation is performed on the initial mask after the screening operation to obtain a foreground object mask corresponding to the target object.

[0085] As an example, in order to generate a high-quality foreground object mask, the mask preprocessing submodule also performs a series of image preprocessing operations, including hole filling, to make the mask more complete by filling the holes in the target object.

[0086] As an example, the mask preprocessing submodule also performs image smoothing, thereby reducing noise and irregularities of the mask to make the mask edge smoother and more accurate.

[0087] As an example, through the above screening based on preset evaluation indicators and / or performing preset preprocessing operations, preprocessed high-quality foreground object masks can be obtained. These masks can effectively distinguish the target object from the background and provide accurate target object information for the subsequent three-dimensional reconstruction process.

[0088] The step of performing a screening operation on the initial mask based on a preset evaluation index to obtain a foreground object mask corresponding to the target object includes:

[0089] Step S131, performing a screening operation on the initial mask based on a preset evaluation index to obtain an intermediate mask;

[0090] Step S132: Fusing the intermediate mask with the corresponding image frame to obtain a foreground object mask corresponding to the target object.

[0091] In this embodiment, after the initial mask is subjected to a screening operation based on a preset evaluation index and / or a preset preprocessing operation, an intermediate mask is obtained. After obtaining the intermediate mask, the intermediate mask is also fused with the corresponding image frame instead of just directly using the intermediate mask (this is because the image frame contains more information), thereby avoiding missing important information, providing accurate target object information for subsequent three-dimensional reconstruction, and improving the quality of the reconstruction result.

[0092] Step S20, selecting a key frame from the input image sequence, and determining the camera pose information of the key frame corresponding to the viewing angle;

[0093] In this embodiment, key frames are selected from the input image sequence instead of using the entire input image sequence to perform subsequent reconstruction of the target object.

[0094] In this embodiment, it should be noted that the camera pose information corresponding to the viewing angle of each key frame may be different.

[0095] As an example, in this embodiment, a group of key frames are selected from the input image sequence based on a preset frame extraction interval and mask generation effect.

[0096] As an example, in this embodiment, an accurate mask and posture of the target object are provided for the subsequent three-dimensional reconstruction process.

[0097] As an example, in the pose calibration stage, pose calibration can be performed based on a sparse 3D reconstruction module of the COLMAP algorithm.

[0098] As an example, the camera pose is calculated based on the disparity information between key frames.

[0099] By rationally selecting key frames and accurate pose calibration, the system can improve the accuracy and stability of reconstruction, thereby achieving more accurate image processing results.

[0100] Step S30: reconstructing a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information.

[0101] In this embodiment, the target object reconstruction model is mainly implemented based on neural implicit surface representation, which includes a series of steps such as initializing differentiable marching tetrahedrons, training neural volume representation, generating mesh topology structure, implicit texture and lighting representation, refined mesh model generation, and differentiable light rendering.

[0102] In the above-mentioned neural implicit surface representation process, the gradient is calculated through pixel loss optimization and the network parameters are optimized, and finally the generation of a refined single-target three-dimensional model can be completed.

[0103] The present application provides an image processing method, device and storage medium. Compared with the related art in which the three-dimensional reconstruction technology based on a monocular camera reconstructs objects in a real scene, it is difficult to meet the corresponding reconstruction efficiency requirements. In the present application, an input image sequence of a target object is obtained, and a mask image of the target object is determined based on the image frame of the input image sequence; a key frame is selected from the input image sequence, and the camera pose information of the key frame corresponding to the viewing angle is determined; based on the key frame, the mask image and the camera pose information, a three-dimensional model of the target object is reconstructed. In the present application, by generating a mask image of the target object and extracting important information of the target object through key frame selection (including the camera pose information of the corresponding viewing angle), the three-dimensional model of the target object is reconstructed. It can be understood that the above process does not require pre-calibration of mask images for training, avoids the need for a large number of manual data preprocessing processes, reduces noise and unnecessary calculations in the reconstruction process, and because the present application does not need to introduce new sensors, important information of the target object can be extracted, and the reconstruction of the three-dimensional model is realized, which saves costs, that is, in this embodiment, the reconstruction efficiency of the corresponding target object is improved.

[0104] Further, based on the first embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the step of selecting a key frame from the input image sequence and determining the camera pose information of the viewing angle corresponding to the key frame includes:

[0105] Step S21, selecting key frames from the input image sequence according to preset selection factors, wherein the preset selection factors include a preset frame extraction interval and a mask generation effect;

[0106] Step S22, performing feature extraction, feature matching and camera pose solving on the selected key frame to obtain the camera pose information under the corresponding viewing angle of the key frame, wherein the camera pose information includes the camera intrinsic parameter matrix and extrinsic parameter matrix.

[0107] As an example, the selection of key frames is usually based on the following factors: frame extraction interval, mask generation effect, image quality of key frames, motion status between corresponding frames, etc.;

[0108] As an example, key frames are selected based on the frame extraction interval and the mask generation effect (including the quality of mask generation). After selecting the key frames, the position and direction of the camera in the three-dimensional space are inferred based on the known camera parameters and the two-dimensional coordinates of the feature points, and finally the image pose calibration is completed.

[0109] In this embodiment, feature extraction, feature matching and camera pose solution are performed on the selected key frames to obtain the camera pose information under the corresponding viewing angle of the key frames, wherein the camera pose information includes the camera intrinsic parameter matrix and the extrinsic parameter matrix.

[0110] As an example, when calibrating the key frame pose, the COLMAP algorithm is used to perform feature extraction, feature matching and camera pose solution on the selected key frames to obtain the camera intrinsic parameter and extrinsic parameter matrix of the input image, which is matched with the input image as the input of the image processing algorithm.

[0111] In the present application, key frames are selected, and feature extraction, feature matching and camera pose solution are performed on the selected key frames to obtain the camera pose information under the corresponding viewing angle of the key frames. Therefore, the amount of processed data and unnecessary calculations can be reduced, thereby speeding up image processing and improving the real-time and efficiency of the system.

[0112] Furthermore, based on the above-mentioned embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the three-dimensional model of the target object can also be reconstructed by means of multiple views.

[0113] Reconstructing the three-dimensional model of the target object by means of multiple views may specifically be:

[0114] After taking a photo of the target object, a sparse point cloud of the target object is generated, and a corresponding dense point cloud is generated based on the sparse point cloud of the target object. Then, texture mapping is performed on the target object based on the corresponding dense point cloud, thereby achieving three-dimensional reconstruction of the target object.

[0115] Furthermore, the step of reconstructing a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information includes:

[0116] Step S30, reconstructing a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information by presetting a single target model of the neural implicit surface.

[0117] As an example, a single-objective model with a neural implicit surface is supervised based on a loss in image space and performs gradient-based optimization for all stages.

[0118] As an example, by presetting a single target model of a neural implicit surface, based on the key frame, the mask image and the camera pose information, the process of reconstructing a three-dimensional model of the target object is as follows: Figure 4 shown.

[0119] As an example, according to the key frame RGB image, target mask, and calibrated key frame corresponding view camera pose information, a high-precision three-dimensional model of the target object is generated using a neural implicit surface-based three-dimensional reconstruction algorithm (with a preset neural implicit surface single target model).

[0120] As an example, a single-objective model of a preset neural implicit surface learns topology, material, texture, and ambient light from keyframe images (2D images) to jointly generate a three-dimensional model.

[0121] As an example, the topology of a triangle mesh is directly optimized using differentiable marching tetrahedra.

[0122] As an example, material, texture gloss is learned through volume rendering and efficiently encoded using an MLP network with position encoding. As a result, the output representation is a 2D texture map and triangular surface mesh file with spatial variations.

[0123] As an example, 2D texture maps and triangle surface mesh files can be used directly in the Blender engine.

[0124] As an example, the step of reconstructing a three-dimensional model of the target object by presetting the single target model of the neural implicit surface based on the key frame, the mask image and the camera pose information includes:

[0125] Step S301, generating a rendering image at a corresponding viewing angle based on the key frame, the mask image and the camera pose information by presetting a single target model of a neural implicit surface;

[0126] As an example, the topology of a triangle mesh is directly optimized using differentiable marching tetrahedra.

[0127] As an example, a keyframe RGB image is input and an implicit SDF field representation model is used (using a neural SDF representation can be used as a smoothness prior to generate smoother shapes). After initializing the differentiable marching tetrahedron, the neural volume representation is trained to predict the SDF values ​​of the tetrahedron vertices (as an example, directly optimizing the vertex-by-vertex attributes can capture higher frequency details and is faster to train), and then a mesh topology is generated and the mesh topology is rendered differently;

[0128] As an example, the implicit SDF field is converted to a triangle mesh using a differentiable marching tetrahedron layer (MT).

[0129] As an example, the shape represented by the implicit (SDF) field is converted to a mesh in the post-processing step using the Marching Cubes (MC) algorithm. However, using the MC algorithm will introduce discretization errors. Based on this, the triangle mesh is explicitly rendered during the optimization process and the mesh is extracted using Deep Marching Tetrahedra (DMTet) through differentiable rendering.

[0130] As an example, DMTet is a hybrid 3D representation that defines the shape of a discrete SDF at the vertices of a deformable tetrahedral mesh.

[0131] During training, the loss computed from renderings of the 3D model is back-propagated to the implicit SDF field to update the model's surface topology, enabling direct end-to-end optimization of both surface mesh and rendered appearance.

[0132] As an example, the step of generating a rendered image at a corresponding viewing angle based on the key frame, the mask image and the camera pose information by presetting the single target model of the neural implicit surface includes:

[0133] Step B1, predicting a mesh topology structure corresponding to a key frame based on the key frame, the mask image, the camera pose information, and a preset differentiable marching tetrahedron;

[0134] As an example, based on the key frame, the mask image and the camera pose information, a tetrahedral mesh with a vertex position v is determined, and then, the SDF value s and the deformation vector Δv are learned based on DMTet.

[0135] As an example, SDF values ​​can be stored as the value of each mesh vertex, and the SDF field can be implicitly stored by the neural network.

[0136] In this embodiment, in each optimization step, the SDF values ​​are first converted to a triangular surface mesh using a marching tetrahedron layer, and the topology of the stored surface is changed in the DMTet by changing the SDF values ​​of the mesh vertices.

[0137] As an example, a differentiable rasterizer is subsequently used to render the mesh to produce an output image, and the image space loss gradient is back-propagated to the SDF value and the offset value (represented by the network weights), thereby obtaining the corresponding mesh topology.

[0138] The step of predicting the mesh topology structure corresponding to the key frame based on the key frame, the mask image, the camera pose information and the preset differentiable marching tetrahedron includes:

[0139] Step C1, predicting SDF values ​​of tetrahedron vertices based on the key frame, the mask image, the camera pose information, and a preset differentiable marching tetrahedron;

[0140] Step C2, performing a preset regularization process on the SDF value to obtain a processed SDF value;

[0141] Step C3: predicting the mesh topology corresponding to the key frame based on the processed SDF value.

[0142] To reduce floating objects and internal geometry, we regularize the SDF value of DMTet. Given the binary cross entropy H, the sigmoid function σ and the sign function sign(x), the regularizer is defined as

[0143] Among them, sign(s i )≠sign(s j ), in this embodiment, the number of symbol flips is reduced, the surface is simplified, and the internal geometry and floating are smoothed.

[0144]

[0145]

[0146] Step B2: predicting the image texture, normal texture and lighting information of the mesh corresponding to the key frame based on the mesh topology, and generating a rendered image under the corresponding viewing angle.

[0147] In this embodiment, after the mesh topology structure is obtained, the image texture, normal texture and illumination information of the mesh corresponding to the key frame are predicted based on the mesh topology structure to generate a rendered image at a corresponding viewing angle.

[0148] In this embodiment, it should be clarified that texture refers to the texture of the surface mesh. Currently, texture parameterization is an active research field in computer graphics.

[0149] In this embodiment, in order to optimize the topology, the model needs to continuously update parameterization, however, the discontinuity of texture representation may affect the training process. In order to robustly handle textures during topology optimization, volume textures are used in this embodiment and added to the texture representation using world space position indexes to ensure that the map changes smoothly with vertex translation and topology changes.

[0150] In this embodiment, the memory footprint of the volume texture grows cubically, which is unmanageable for the required target resolution. For this reason, in this embodiment, a multi-layer perceptron (MLP) is used to encode all material parameters into a compact representation (this representation can adaptively distribute details around the 2D manifold representing the surface mesh).

[0151] In this embodiment, an image-based lighting model is used to represent illumination;

[0152] As an example, the scene ambient light is provided by a high-resolution cube map, and ω can be calculated by the following formula o The outgoing radiation L(ω o ):

[0153] L(ω o )=∫ Ω L i (ω i )f(ω i ,ω o )(ω i ·n)dω i

[0154] Specifically, the outgoing radiation is from the direction ω i The incident radiation L i (ω i )Li(ωi) and BSDF f(ω i ,ω o ), where the integration domain is the hemisphere Ω around the surface intersection normal n. For the specular part of the outgoing radiation, a BSDF-based microfacet specular shadow model can be used, which is calculated as follows:

[0155]

[0156] Where D, G, and F are functions representing the GGX normal distribution (NDF), geometric attenuation, and Fresnel term, respectively.

[0157] In this embodiment, a high-quality estimate of image-based illumination can be obtained by Monte Carlo integration, where spherical Gaussian (SG) and spherical harmonics (SH) are common approximations of image-based illumination, allowing the illumination frequency to be controlled by changing the number of SG lobes (or SH coefficients), and are effective representations of low- to mid-frequency illumination.

[0158] However, many SG lobes are required to represent high-frequency and high-specular materials, which brings high running costs and impairs training stability. Since segmentation and approximation is a popular and effective illumination method based on full-frequency images, the illumination integral in this embodiment can be approximated as:

[0159] L(ω o )≈∫ Ω f(ω i ,ω o )(ω i ·n)dω i ∫ Ω L i (ω i )D(ω i ,ω o )(ω i ·n)dω i

[0160] The first term represents the integration of the specular BSDF with a solid white ambient light, which depends only on the parameters cosθ=ωi·n and the roughness r of the BSDF, and can be pre-computed and stored in a 2D lookup texture. The second term represents the integration of the incident radiance with the specular NDF. This term is represented by the filter cube map through pre-integration.

[0161] In this embodiment, a differentiable version of the segmentation and shading model is introduced to learn the environment lighting from image observations through differentiable rendering. Specifically, the texels of the cube map (resolution 6×512×512) can be used as trainable parameters.

[0162] Step S302, determining loss information of the rendered image and the key frame;

[0163] Step S303, updating network parameters of a single-objective model of a preset neural implicit surface based on the loss information;

[0164] In this embodiment, based on the loss information of the rendered image and the key frame, the network parameters of the single-target model of the preset neural implicit surface are updated. Specifically, the network parameters may be image texture, normal texture, lighting information, etc.

[0165] Step S304, returns the single target model of the updated preset neural implicit surface, and generates a rendering image under the corresponding perspective based on the key frame, the mask image and the camera pose information, until the number of updates reaches the preset number, to obtain the three-dimensional model of the target object.

[0166] In this embodiment, if the reconstructed three-dimensional model of the target object does not meet the requirements, such as Figure 5 As shown, the single target model of the preset neural implicit surface after the update is returned, and the step of generating a rendering image under the corresponding viewing angle based on the key frame, the mask image and the camera pose information is performed until the number of updates reaches the preset number of times, thereby obtaining the three-dimensional model of the target object. The preset number of times can be set.

[0167] In the present application, based on the loss information, the network parameters of the single-target model of the preset neural implicit surface are updated to achieve accurate reconstruction of the target object.

[0168] Reference Figure 3 , Figure 3 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0169] like Figure 3 As shown, the image processing device may include: a processor 1001 , a memory 1005 , and a communication bus 1002 . The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005 .

[0170] Optionally, the image processing device may further include a user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, a WiFi module, etc. The user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. The network interface may include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0171] Those skilled in the art will understand that Figure 3 The structure of the image processing device shown in the figure does not constitute a limitation on the image processing device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0172] like Figure 3As shown, the memory 1005 as a storage medium may include an operating system, a network communication module and an image processing program. The operating system is a program that manages and controls the hardware and software resources of the image processing device, and supports the operation of the image processing program and other software and / or programs. The network communication module is used to realize the communication between the cameras inside the memory 1005, and the communication with other hardware and software in the device.

[0173] exist Figure 3 In the image processing device shown, the processor 1001 is used to execute the image processing program stored in the memory 1005 to implement the steps of any one of the above-mentioned image processing methods.

[0174] The specific implementation of the image processing device of the present application is basically the same as the above-mentioned embodiments of the image processing method, and will not be repeated here.

[0175] The present application also provides an image processing device, the device comprising:

[0176] A mask generation module, used for acquiring an input image sequence of a target object, and determining a mask image of the target object based on image frames of the input image sequence;

[0177] A key frame selection and posture calibration module, used to select a key frame from the input image sequence and determine the camera posture information of the key frame corresponding to the viewing angle;

[0178] A three-dimensional reconstruction module is used to reconstruct a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information.

[0179] In a possible implementation manner of the present application, the image processing device is used to implement:

[0180] Allocating each pixel in the image frame to a different object instance by using a preset instance segmentation network structure;

[0181] Determine the target object from the object instance, distinguish the target object and the background in the image frame, and obtain an initial mask of the target object;

[0182] The initial mask is screened based on preset evaluation indicators to obtain a foreground object mask corresponding to the target object, wherein the preset evaluation indicators include an intersection-over-union ratio, a screen-to-body ratio, and a distance from a local center area of ​​the target object in the image frame.

[0183] In a possible implementation manner of the present application, the image processing device is used to implement:

[0184] Performing a screening operation on the initial mask based on a preset evaluation index to obtain an intermediate mask;

[0185] The intermediate mask is fused with the corresponding image frame to obtain a foreground object mask corresponding to the target object.

[0186] In a possible implementation manner of the present application, the image processing device is used to implement:

[0187] Performing a screening operation on the initial mask based on a preset evaluation index;

[0188] A preset preprocessing operation is performed on the initial mask after the screening operation to obtain a foreground object mask corresponding to the target object.

[0189] In a possible implementation manner of the present application, the image processing device is used to implement:

[0190] Selecting key frames from the input image sequence according to preset selection factors, wherein the preset selection factors include a preset frame extraction interval and a mask generation effect;

[0191] The selected key frames are subjected to feature extraction, feature matching and camera pose solution to obtain the camera pose information under the corresponding viewing angle of the key frames, wherein the camera pose information includes the camera intrinsic parameter matrix and extrinsic parameter matrix.

[0192] In a possible implementation manner of the present application, the image processing device is used to implement:

[0193] By presetting a single target model of a neural implicit surface, a three-dimensional model of the target object is reconstructed based on the key frame, the mask image and the camera pose information.

[0194] In a possible implementation manner of the present application, the image processing device is used to implement:

[0195] By presetting a single target model of a neural implicit surface, based on the key frame, the mask image and the camera pose information, a rendering image under a corresponding viewing angle is generated;

[0196] Determining loss information of the rendered image and the key frame;

[0197] Based on the loss information, updating network parameters of a single-objective model of a preset neural implicit surface;

[0198] Return the single target model of the updated preset neural implicit surface, and generate a rendering image under the corresponding perspective based on the key frame, the mask image and the camera pose information until the number of updates reaches a preset number, thereby obtaining a three-dimensional model of the target object.

[0199] In a possible implementation manner of the present application, the image processing device is used to implement:

[0200] Based on the key frame, the mask image, the camera pose information, and a preset differentiable marching tetrahedron, predicting a mesh topology structure corresponding to the key frame;

[0201] Based on the mesh topology, the image texture, normal texture and lighting information of the mesh corresponding to the key frame are predicted to generate a rendered image at a corresponding viewing angle.

[0202] In a possible implementation manner of the present application, the image processing is used to achieve:

[0203] Based on the key frame, the mask image, the camera pose information, and a preset differentiable marching tetrahedron, predicting SDF values ​​of tetrahedron vertices;

[0204] Performing a preset regularization process on the SDF value to obtain a processed SDF value;

[0205] Based on the processed SDF value, the mesh topology corresponding to the key frame is predicted.

[0206] The specific implementation of the image processing device of the present application is basically the same as the above-mentioned embodiments of the image processing method, and will not be repeated here.

[0207] An embodiment of the present application provides a storage medium, and the storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned image processing methods.

[0208] The specific implementation of the storage medium of the present application is basically the same as the above-mentioned embodiments of the image processing method, and will not be repeated here.

[0209] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0210] The specific implementation of the computer program product of the present application is basically the same as the above-mentioned embodiments of the image processing method, and will not be repeated here.

[0211] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0212] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0213] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus hardware platform, or by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes several instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0214] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an input image sequence of a target object, and determine a mask image of the target object based on image frames of the input image sequence; Selecting a key frame from the input image sequence, and determining the camera pose information of the key frame corresponding to the viewing angle; Based on the key frame, the mask image and the camera pose information, a three-dimensional model of the target object is reconstructed.

2. The image processing method according to claim 1, characterized in that: The step of determining the mask image of the target object based on the image frames of the input image sequence comprises: Allocating each pixel in the image frame to a different object instance by using a preset instance segmentation network structure; Determine the target object from the object instance, distinguish the target object and the background in the image frame, and obtain an initial mask of the target object; The initial mask is screened based on preset evaluation indicators to obtain a foreground object mask corresponding to the target object, wherein the preset evaluation indicators include an intersection-over-union ratio, a screen-to-body ratio, and a distance from a local center area of ​​the target object in the image frame.

3. The image processing method according to claim 2, characterized in that: The step of performing a screening operation on the initial mask based on a preset evaluation index to obtain a foreground object mask corresponding to the target object includes: Performing a screening operation on the initial mask based on a preset evaluation index to obtain an intermediate mask; The intermediate mask is fused with the corresponding image frame to obtain a foreground object mask corresponding to the target object.

4. The image processing method according to claim 2, wherein: The step of performing a screening operation on the initial mask based on a preset evaluation index to obtain a foreground object mask corresponding to the target object includes: Performing a screening operation on the initial mask based on a preset evaluation index; A preset preprocessing operation is performed on the initial mask after the screening operation to obtain a foreground object mask corresponding to the target object.

5. The image processing method according to claim 1, wherein: The step of selecting a key frame from the input image sequence and determining the camera pose information of the key frame corresponding to the viewing angle comprises: Selecting key frames from the input image sequence according to preset selection factors, wherein the preset selection factors include a preset frame extraction interval and a mask generation effect; The selected key frames are subjected to feature extraction, feature matching and camera pose solution to obtain the camera pose information under the corresponding viewing angle of the key frames, wherein the camera pose information includes the camera intrinsic parameter matrix and extrinsic parameter matrix.

6. The image processing method according to claim 1, characterized in that: The step of reconstructing a three-dimensional model of the target object based on the key frame, the mask image and the camera pose information comprises: By presetting a single target model of a neural implicit surface, a three-dimensional model of the target object is reconstructed based on the key frame, the mask image and the camera pose information.

7. The image processing method according to claim 6, characterized in that: The step of reconstructing a three-dimensional model of the target object by presetting the single target model of the neural implicit surface based on the key frame, the mask image and the camera pose information comprises: By presetting a single target model of a neural implicit surface, based on the key frame, the mask image and the camera pose information, a rendering image under a corresponding viewing angle is generated; Determining loss information of the rendered image and the key frame; Based on the loss information, updating network parameters of a single-objective model of a preset neural implicit surface; Return the single target model of the updated preset neural implicit surface, and generate a rendering image under the corresponding perspective based on the key frame, the mask image and the camera pose information until the number of updates reaches a preset number, thereby obtaining a three-dimensional model of the target object.

8. The image processing method according to claim 7, characterized in that: The step of generating a rendered image at a corresponding viewing angle based on the key frame, the mask image and the camera pose information by presetting the single target model of the neural implicit surface comprises: Based on the key frame, the mask image, the camera pose information, and a preset differentiable marching tetrahedron, predicting a mesh topology structure corresponding to the key frame; Based on the mesh topology, the image texture, normal texture and lighting information of the mesh corresponding to the key frame are predicted to generate a rendered image at a corresponding viewing angle.

9. The image processing method according to claim 8, characterized in that: The step of predicting the mesh topology structure corresponding to the key frame based on the key frame, the mask image, the camera pose information and the preset differentiable marching tetrahedron includes: Based on the key frame, the mask image, the camera pose information, and a preset differentiable marching tetrahedron, predicting SDF values ​​of tetrahedron vertices; Performing a preset regularization process on the SDF value to obtain a processed SDF value; Based on the processed SDF value, the mesh topology corresponding to the key frame is predicted.

10. An image processing device, characterized in that: The invention comprises a memory, a processor and an image processing program stored in the memory and executable on the processor, wherein the processor implements the steps of the image processing method according to any one of claims 1 to 9 when executing the image processing program.

11. A storage medium, characterized in that: The storage medium stores an image processing program, and when the image processing program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 9 are implemented.

Images