Object reconstruction method and device

Abstract

The invention provides an object reconstruction method and device, and the method comprises the steps: obtaining a multispectral image of a shooting scene, the shooting scene comprises a target object, and the multispectral image indicates the multispectral information of the space of the shooting scene; acquiring spatial data of the target object, wherein the spatial data indicates two-dimensional spatial information or three-dimensional spatial information of the target object; obtaining the material of the target object based on the multispectral image and the spatial data of the target object; and obtaining a reconstruction result of the target object based on the spatial data of the target object and the material of the target object, the reconstruction result comprising a two-dimensional view or a three-dimensional view of the target object. According to the method, the material of the target object is obtained through multispectral image recognition, the material is introduced in the target object reconstruction process, and the two-dimensional reconstruction effect or the three-dimensional reconstruction effect of the target object is improved.

Description

Technical Field

[0001] This application relates to the field of optical technology, and in particular to an object reconstruction method and apparatus. Background Technology

[0002] Multispectral array devices can directly capture the spatial spectral information of a scene being photographed. By comparing this information with a spectral library, they can identify the material of objects at each pixel location in space, providing more accurate object information for downstream applications. Multispectral array applications are also an important trend in the development of next-generation edge devices.

[0003] Existing 3D reconstruction and editing systems generally utilize traditional single-view or multi-view images (such as RGB images, infrared images, depth images, etc.) or point cloud data to obtain a 3D representation of a spatial target, including display representations such as point clouds, voxels, and meshes, as well as implicit representations such as neural radiance fields (NeRF). Then, rendering and reconstruction are performed based on this 3D representation to obtain a 3D view. However, the 3D views reconstructed using existing technologies are of poor quality and lack realism. Summary of the Invention

[0004] The embodiments of this application provide an object reconstruction method that obtains the material of the target object through multispectral image recognition, introduces the material during the target object reconstruction process, and improves the three-dimensional reconstruction effect or two-dimensional reconstruction effect of the target object.

[0005] In a first aspect, this application provides an object reconstruction method, comprising: acquiring a multispectral image of a shooting scene (also referred to as a target shooting scene, i.e., the shooting scene where the target object to be reconstructed is located), wherein the target shooting scene includes a target object (i.e., the object to be reconstructed), and the multispectral image indicates the multispectral information of the space of the target shooting scene; acquiring spatial data of the target object, wherein the spatial data indicates the two-dimensional spatial information or three-dimensional spatial information of the target object; obtaining the material of the target object based on the multispectral image and the spatial data of the target object; and obtaining the reconstruction result of the target object based on the spatial data and the material of the target object, wherein the reconstruction result includes a two-dimensional view or a three-dimensional view of the target object.

[0006] The object reconstruction method provided in this application obtains the material of a 2D or 3D instance of a target object (hereinafter referred to as the target 2D or 3D instance) through multispectral image recognition, and then uses the optical properties (such as reflectivity, gloss, etc.) corresponding to the material of the target 2D or 3D instance to render the target 2D or 3D instance, thereby improving the two-dimensional or three-dimensional reconstruction effect of the target object. For example, the reconstructed two-dimensional or three-dimensional view has more accurate and realistic colors.

[0007] It is readily understood that the relationship between data and information mentioned in the embodiments of this application is that information refers to the information content that the data expresses or carries after processing. For example, the spatial data of a target object can be the point cloud data of the target object. By analyzing and processing the point cloud data of the target object, the distribution of the target object's surface in three-dimensional space can be obtained. The distribution of the target object's surface in three-dimensional space is the three-dimensional spatial information of the target object.

[0008] In one possible implementation, the spatial data of the target object includes the three-dimensional representation data of the target object; a specific implementation of obtaining the material of the target object based on the multispectral image and the spatial data of the target object is as follows: perform instance segmentation on the multispectral image to obtain at least one 2D instance; obtain the material of each 2D instance based on the reflectance spectrum of each 2D instance in the at least one 2D instance; perform instance segmentation on the three-dimensional representation data to obtain a 3D instance of the target object; and obtain the material of the target object based on the material of each 2D instance and the 3D instance of the target object.

[0009] The object reconstruction method of this application is applied to the 3D reconstruction of the target object. By processing the acquired multispectral image and the spatial data of the target object, the material of the target object can be obtained, so as to introduce the material in the subsequent 3D reconstruction process of the target object and improve the 3D reconstruction effect of the target object.

[0010] Optionally, the three-dimensional spatial information of the target object can be expressed in various ways, such as explicit expression like point cloud, voxel, and mesh, and implicit expression like neural radiation field. In other words, the three-dimensional expression data can be any of the three-dimensional expression data among three-dimensional point cloud data, three-dimensional voxel data, three-dimensional mesh data, or neural radiation field data. This application does not limit the three-dimensional expression method of the three-dimensional expression data, and a suitable expression method can be selected as needed.

[0011] The object reconstruction method provided in this application is illustrated below, using three-dimensional point cloud data as an example of three-dimensional representation data.

[0012] In another possible implementation, a specific implementation of obtaining the material of each 2D instance based on the reflection spectrum of each 2D instance in at least one 2D instance is as follows: based on the reflection spectrum of each 2D instance and the spectrum of ambient light, the spectral reflectance of each 2D instance is obtained, where the spectrum of ambient light is the spectrum of ambient light of the target shooting scene; based on the spectral reflectance of each 2D instance, the material of each 2D instance is obtained by retrieving from a material spectral library, where the material spectral library records multiple materials and the spectral reflectance corresponding to each material.

[0013] In another possible implementation, a specific implementation of obtaining the material of each 2D instance based on the reflection spectrum of each 2D instance in at least one 2D instance is as follows: Based on the reflection spectrum of each 2D instance, the spectrum of ambient light, and the spectrum of the target standard light source, the spectral reflectance of each 2D instance is obtained, where the spectrum of ambient light is the spectrum of the ambient light of the target shooting scene; Based on the spectral reflectance of each 2D instance, the material of each 2D instance is obtained by retrieving it from a material spectral library, where the material spectral library records multiple materials and the spectral reflectance of each material under the target standard light source.

[0014] For example, by dividing the reflectance spectrum of each 2D instance by the spectrum of ambient light and then multiplying it by the spectrum of the target standard light source, the spectral reflectance of each 2D instance under the standard light source can be obtained. Then, by searching the material spectral library, the material of each 2D instance under the target standard light source can be obtained. In this way, the material of each 2D instance can be accurately obtained.

[0015] Optionally, the spectral information of the ambient light in the target shooting scene can be obtained through various methods. For example, the spectral information of the ambient light can be calculated from a multispectral image. For instance, firstly, based on the multispectral image, the scene reflectance spectrum is obtained, indicating the spectral distribution of the reflectance of the target shooting scene. Then, based on the scene reflectance spectrum and k standard light sources, the spectrum of the ambient light is obtained, where k is a positive integer greater than 1. Specifically, the scene reflectance spectrum of the actual shooting scene is obtained by inverse kinematics of the multispectral image. Then, based on the scene reflectance spectrum, a fitting operation is performed on each of the K standard light sources to obtain K fitting coefficients. These K fitting coefficients are then used as weights to weight the spectra of the K standard light sources to obtain the spectrum of the ambient light. Alternatively, the scene reflectance spectrum of the actual shooting scene can be obtained by inverse kinematics of the multispectral image. Then, the scene reflectance spectrum is analyzed to determine the component proportion of each standard light source in the scene reflectance spectrum. Finally, the component proportions of each standard light source are used as weighting coefficients to weight and sum the K standard light sources to obtain the spectrum of the ambient light. This eliminates the need for additional hardware for measuring ambient light sources, reducing hardware costs.

[0016] For example, the ambient light spectrum can be obtained by directly measuring the spectrum of the ambient light in the target shooting scene. For instance, the ambient light in the target shooting scene can be measured directly using a spectral measuring instrument, and the measurement result can be used as the spectral data of the ambient light in the target shooting scene. In this way, the light source data of the ambient light can be obtained directly by measuring the ambient light spectrum, without the need for processing multispectral data, thus reducing computing power overhead.

[0017] For example, multi-view RGB images can be used to assist multispectral image analysis to obtain a more accurate spectrum of ambient light in the target shooting scene. Furthermore, multi-view RGB images can be used to calculate the 3D point cloud data of the target object. For instance, by processing RGB image data to obtain foreground and background images, and then using these images to assist multispectral image analysis to obtain the spectrum of ambient light in the target shooting scene, the spectral information of the ambient light in the target shooting scene can be estimated more accurately.

[0018] In another possible implementation, a specific way to obtain the material of the target object based on the material of each 2D instance and the 3D instance of the target object is as follows: perform a registration operation on each 2D instance and the 3D instance of the target object, matching the spatial pixels of each 2D instance to the 3D instance of the target object; and obtain the material of the target object based on the material corresponding to the spatial pixels matched by the 3D instance of the target object.

[0019] For example, multispectral images are acquired by a multispectral sensor capturing the target scene, while spatial data is obtained by using RGB cameras in different poses to capture RGB images of the target scene from different perspectives. Image registration is performed using calibration parameters of the multispectral sensor and RGB cameras (e.g., the poses of the multispectral sensor and the RGB cameras) to match the spatial pixels of each 2D instance to the corresponding 3D instance of the target object. Each pixel in the 2D instance carries material information. For any given 3D instance, the material of the matched pixels is determined by a voting system. For instance, if the ratio of iron, aluminum, and copper in the matched pixels of a 3D instance is 3:2:1, and iron pixels are the most numerous in that 3D instance, then the material of that 3D instance is determined to be iron. This method accurately obtains the material of each 3D instance, which in turn yields the material of each target object.

[0020] Optionally, after determining the material of the target object using the above algorithm, the material name of the target object can be directly output to subsequent steps so that the material can be used to improve the 3D reconstruction effect of the target object. For example, if the material of target object A is determined to be aluminum by the above algorithm, then target object A-aluminum can be directly output; alternatively, the material ID can be output. For example, if the material of target object A is determined to be aluminum by the above algorithm, then the ID corresponding to aluminum can be output as 423, and the output can be represented as target object A-423. In this embodiment, after the material of the target object is determined by the algorithm, the material can be represented as information that uniquely identifies or indicates the material, such as the material ID, code, or a certain attribute of the material, without limiting the specific expression form of the material.

[0021] In another possible implementation, a specific way to obtain the reconstruction result of the target object based on the spatial data and material of the target object is to render the target object based on the 3D instance corresponding to the target object, the material of the target object and the spectrum of ambient light to obtain a three-dimensional view of the target object.

[0022] By introducing materials into 3D instance rendering, the appearance attributes of the target object corresponding to the material are obtained using the material, such as the gloss and reflectivity of the target object. The reflectivity is used to obtain the reflection spectrum of the target object under ambient light. Then, the gloss and reflection spectrum are used to render the 3D instance, thereby improving the 3D rendering effect.

[0023] In another possible implementation, a specific approach to obtaining the reconstruction result of the target object based on its spatial data and material is as follows: 3D point cloud data is used as input to a 3D Gaussian model, outputting a corresponding 3D Gaussian point cloud. Each point in the 3D Gaussian point cloud includes material information (this can also be described as each point in the Gaussian point cloud carrying material information, indicating the material of the entity point on the target object corresponding to each point in the Gaussian point cloud). The color representation parameters of the 3D Gaussian model are determined based on the material of the target object. Based on the 3D Gaussian point cloud and the spectrum of ambient light, the target object is rendered to obtain a 3D view of the target object. Processing the 3D point cloud data using a 3D Gaussian model and then using the processed 3D Gaussian point cloud for rendering further improves the 3D rendering effect.

[0024] In another possible implementation, a specific approach to obtaining spatial data of the target object is to acquire images of the target shooting scene from multiple different perspectives; and based on these images, obtain the three-dimensional point cloud data of the target object.

[0025] There are various methods for acquiring images from multiple different perspectives. For example, multiple RGB cameras in different poses can be used to capture images of the target scene, resulting in multiple RGB images from different perspectives. Alternatively, depth and RGB images from different perspectives can be acquired using depth and RGB cameras in different poses. Or, an RGB image of the target scene from one perspective can be acquired using an RGB camera in a different pose than the multispectral sensor, while the RGB image from the other perspective is converted from the multispectral image. This allows the multispectral image to participate in the acquisition of spatial data, and only one RGB camera is needed to obtain RGB images from two perspectives, reducing hardware costs.

[0026] Secondly, this application also provides an object editing method, which obtains a two-dimensional view or a three-dimensional view of a target object, the two-dimensional view or the three-dimensional view of the target object being reconstructed based on the object reconstruction method as described in the first aspect; and edits the two-dimensional view or the three-dimensional view in response to editing parameters input by the user.

[0027] In one possible implementation, the editing parameters include one or more of the following: ambient light editing parameters, 3D instance editing parameters, and viewpoint editing parameters; wherein, the ambient light editing parameters include ambient light parameters, ambient light maps, or ambient light model parameters, and the ambient light parameters include one or more of the following: ambient light spectral parameters, illumination angle parameters, and illumination position parameters; the 3D instance editing parameters include one or more of the following: 3D instance position parameters, 3D instance size parameters, 3D instance material parameters, 3D instance modification parameters, 3D instance addition parameters, and 3D instance deletion parameters; and the viewpoint editing parameters include one or more of the following: viewpoint adjustment parameters, view zoom-out parameters, and view zoom-in parameters.

[0028] The object editing method provided in this application links 3D reconstruction with materials, prompts the effect of 3D reconstruction, and provides editing functions for ambient light information and materials of 3D instances, thereby improving the accuracy and flexibility of 3D editing.

[0029] Thirdly, this application also provides an object reconstruction apparatus, which includes a first acquisition module, a second acquisition module, a first determination module, and a reconstruction module. The first acquisition module acquires a multispectral image of a target shooting scene, the target shooting scene including a target object, and the multispectral image indicates the multispectral information of the space of the target shooting scene. The second acquisition module acquires spatial data of the target object, the spatial data indicating two-dimensional or three-dimensional spatial information of the target object. The first determination module determines the material of the target object based on the multispectral image and the spatial data of the target object. The reconstruction module obtains a reconstruction result of the target object based on the spatial data and the material of the target object, the reconstruction result including a two-dimensional or three-dimensional view of the target object.

[0030] In one possible implementation, the spatial data of the target object includes three-dimensional representation data of the target object; the first determining module is specifically used to: perform instance segmentation on the multispectral image to obtain at least one 2D instance; obtain the material of each 2D instance based on the reflectance spectrum of each 2D instance in the at least one 2D instance; perform instance segmentation on the three-dimensional representation data to obtain a 3D instance of the target object; and obtain the material of the target object based on the material of each 2D instance and the 3D instance of the target object.

[0031] In another possible implementation, a specific implementation of obtaining the material of each 2D instance based on the reflection spectrum of each 2D instance in at least one 2D instance is as follows: Based on the reflection spectrum of each 2D instance, the spectrum of ambient light, and the spectrum of the target standard light source, the spectral reflectance of each 2D instance is obtained, where the spectrum of ambient light is the spectrum of the ambient light of the target shooting scene; Based on the spectral reflectance of each 2D instance, the material of each 2D instance is obtained by retrieving it from a material spectral library, where the material spectral library records multiple materials and the spectral reflectance of each material under the target standard light source.

[0032] In another possible implementation, the object reconstruction apparatus provided in this application further includes a second determining module, which is used to obtain a scene reflection spectrum based on a multispectral image, the scene reflection spectrum indicating the reflection spectrum distribution of the target shooting scene; and to obtain the spectrum of ambient light based on the scene reflection spectrum and k standard light sources, where k is a positive integer greater than 1.

[0033] In another possible implementation, the second determining module is specifically used to: perform fitting operations on K standard light sources based on the scene reflection spectrum to obtain K fitting coefficients; and use the K fitting coefficients as weights to perform weighting operations on the spectra of the K standard light sources to obtain the spectrum of ambient light.

[0034] In another possible implementation, the spectrum of ambient light is obtained based on spectral measurements of the ambient light in the scene being photographed.

[0035] In another possible implementation, the 3D point cloud data of the target object is obtained based on RGB images from multiple perspectives; the spectrum of ambient light is obtained based on multispectral images and RGB images from multiple perspectives.

[0036] In another possible implementation, the first determining module is specifically used to: perform a registration operation on each 2D instance and the 3D instance of the target object, matching the spatial pixels of each 2D instance to the 3D instance of the target object; and obtain the material of the target object based on the material corresponding to the spatial pixels matched to the 3D instance of the target object.

[0037] In another possible implementation, a specific way to obtain the reconstruction result of the target object based on the spatial data and material of the target object is to render the target object based on the 3D instance corresponding to the target object, the material of the target object and the spectrum of ambient light to obtain a three-dimensional view of the target object.

[0038] In another possible implementation, a specific approach to obtaining the reconstruction result of the target object based on its spatial data and material is as follows: 3D representation data is used as input to a 3D Gaussian model, and a 3D Gaussian point cloud corresponding to the 3D representation data is output. Each point in the 3D Gaussian point cloud includes material information, and the color representation parameters of the 3D Gaussian model are determined based on the material of the target object. Based on the 3D Gaussian point cloud and the spectrum of ambient light, the target object is rendered to obtain a 3D view of the target object.

[0039] In another possible implementation, the second acquisition module is specifically used to: acquire images of the target shooting scene from multiple different perspectives; and obtain three-dimensional representation data of the target object based on the images from multiple different perspectives.

[0040] In another possible implementation, the images from multiple different viewpoints include multiple RGB images from multiple different viewpoints; or, the images from multiple different viewpoints include depth images and RGB images from multiple different viewpoints; or, the images from multiple different viewpoints include multiple RGB images from multiple different viewpoints, wherein a portion of the RGB images from the multiple RGB images from multiple different viewpoints is obtained based on multispectral image conversion.

[0041] Fourthly, this application also provides an object editing device, which includes a third acquisition module and an editing module. The third acquisition module is used to acquire a two-dimensional view or a three-dimensional view of a target object, which is reconstructed based on the object reconstruction method described in the first aspect. The editing module is used to edit the two-dimensional view or the three-dimensional view in response to editing parameters input by the user.

[0042] In one possible implementation, the editing parameters include one or more of the following: ambient light editing parameters, 3D instance editing parameters, and viewpoint editing parameters; wherein, the ambient light editing parameters include ambient light parameters, ambient light maps, or ambient light model parameters, and the ambient light parameters include one or more of the following: ambient light spectral parameters, illumination angle parameters, and illumination position parameters; the 3D instance editing parameters include one or more of the following: 3D instance position parameters, 3D instance size parameters, 3D instance material parameters, 3D instance modification parameters, 3D instance addition parameters, and 3D instance deletion parameters; and the viewpoint editing parameters include one or more of the following: viewpoint adjustment parameters, view zoom-out parameters, and view zoom-in parameters.

[0043] Fifthly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the methods described in the first aspect and / or the second aspect to be implemented.

[0044] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the methods described in the first aspect and / or the second aspect to be implemented.

[0045] In a seventh aspect, embodiments of this application also provide a computer program or computer program product, the computer program or computer program product including instructions that, when executed, cause a computer to perform the methods described in the first aspect and / or the second aspect.

[0046] Eighthly, embodiments of this application also provide a chip including at least one processor and a communication interface, the processor being configured to perform the methods described in the first aspect and / or the second aspect. Attached Figure Description

[0047] Figure 1 A system architecture diagram of a 3D reconstruction system applying an object reconstruction method provided in an embodiment of this application is shown.

[0048] Figure 2 A flowchart illustrating an object reconstruction method provided in an embodiment of this application;

[0049] Figure 3 A rear view of a mobile phone is shown;

[0050] Figure 4 A schematic diagram of a target scene captured by a multispectral camera is shown;

[0051] Figure 5 This illustration shows a schematic diagram of the multispectral image data processing procedure according to an embodiment of this application;

[0052] Figure 6 A schematic diagram illustrates the process of combining 3D instances and multispectral material analysis results to obtain 3D instance material analysis results.

[0053] Figure 7 A flowchart illustrating an object editing method provided in an embodiment of this application is shown;

[0054] Figure 8 This is a schematic diagram of the structure of an object reconstruction device provided in an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the structure of an object editing device provided in an embodiment of this application;

[0056] Figure 10 This diagram illustrates one deployment method of the object reconstruction apparatus and object editing apparatus of this application;

[0057] Figure 11A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0058] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0059] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first RGB image data" and "second RGB image data," etc., are used to distinguish different RGB image data, not to describe a specific order of RGB image data.

[0060] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0061] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0062] To facilitate understanding of the solutions in the embodiments of this application, the technical terms involved in this document will be explained first.

[0063] Multispectral / hyperspectral images: Unlike traditional true-color (RGB) images that only contain three color channels, multispectral or hyperspectral images describe the response characteristics of more spectral bands. The difference between multispectral and hyperspectral images lies in the number of bands and the band spacing; there is no difference in their data representation. In the embodiments of this application, "multispectral images" refers to both multispectral and hyperspectral images without making any special distinction.

[0064] Array multispectral: Multispectral devices or data with spatial dimensions. Compared to single-point multispectral data, it can describe the differences in the spatial distribution of environmental spectra, thus providing a more accurate description of the material properties of objects.

[0065] Spectral Library: A database consisting of the spectral reflectance of typical materials measured under standard light sources.

[0066] Spectral response curve: describes the mapping relationship between two different types of images (generally multispectral images to RGB images) in the spectral dimension.

[0067] Ambient light information: Ambient light information describes the ambient lighting information of the subject. It can be in the form of light source type and position angle, or in the form of ambient light map.

[0068] 3D Instance Materials: Objects made of different materials (such as wood or metal), even if they are the same color, will appear differently under different lighting conditions. These differences can be identified using multispectral data.

[0069] Existing 3D reconstruction and editing systems suffer from poor rendering realism and insufficient editing flexibility. For example, one related technology uses a multi-view, multispectral camera to acquire multiple views of each spectral band, then reconstructs the corresponding 3D point cloud based on these views, and finally merges the reconstructed 3D point clouds from multiple bands into spectral 3D data. This system achieves simple 3D spectral reconstruction based on a multispectral camera, but the rendering effect is poor.

[0070] The second related technology utilizes multispectral images for 3D reconstruction, employing point cloud data and other data to assist in multispectral 3D reconstruction. This system acquires the 3D information of the object through spatial imaging devices such as lasers and structured light, while simultaneously acquiring the multispectral information of the object through a multispectral camera. Finally, based on camera calibration and other information, the 3D and multispectral information are fused to obtain the 3D spectral reconstruction result.

[0071] Similar to related technologies, this system achieves simple 3D spectral reconstruction, but it does not analyze the material of objects, nor does it perform operations such as ambient light editing or 3D instance editing based on the results of material analysis.

[0072] It is evident that the 3D reconstruction and editing systems of related technologies cannot directly and accurately represent the material of objects, cannot accurately identify the material of objects at each location in space when editing objects and environments, cannot provide material editing functions, and have poor rendering realism.

[0073] In view of this, embodiments of this application provide an object reconstruction method, an object editing method, and an apparatus. This method processes multispectral data and spatial data to obtain the material of the target object to be reconstructed. Then, based on the material and the spatial data of the target object, it renders the target object to obtain the reconstruction result, improving the reconstruction effect of the target object. For example, it improves the accuracy of the reconstructed color and the gloss accuracy of the target object, thereby making the presentation effect of the reconstructed target object closer to what the human eye sees, achieving "what you see is what you get".

[0074] The object reconstruction method, object editing method, and apparatus provided in this application can be applied to the three-dimensional reconstruction of a target object to obtain a three-dimensional view of the target object, thereby improving the rendering effect of the three-dimensional view, such as the color accuracy and gloss reproduction of the three-dimensional view, so that the rendered three-dimensional view is close to the effect of the target object seen by the real human eye, realizing "what you see is what you get".

[0075] Of course, the object reconstruction method, object editing method, and apparatus provided in this application can also be applied to the two-dimensional reconstruction of the target object to obtain a two-dimensional view of the target object, thereby improving the color accuracy of the two-dimensional view. The following section uses the three-dimensional reconstruction of an object as an example to describe the object reconstruction method, object editing method, and apparatus provided in this application.

[0076] The following detailed description, with reference to the accompanying drawings, details the specific implementation of an object reconstruction method, object editing method, and apparatus provided in this application.

[0077] Figure 1 A system architecture diagram of a 3D reconstruction system applying an object reconstruction method provided in an embodiment of this application is shown. Figure 1 As shown, the 3D reconstruction system includes a multispectral imaging device, a 3D spatial data acquisition device, an object reconstruction device, a storage device, and a display device. The multispectral imaging device acquires multispectral image data of the target scene, which contains a target object. The 3D spatial data acquisition device acquires the 3D spatial data (e.g., 3D point cloud data) of the target object. The object reconstruction device processes the multispectral image data and 3D spatial data to obtain the material of the target object. Based on the material and 3D instance of the target object, it renders a 3D view of the target object, incorporating the material into the 3D reconstruction process. By utilizing the material properties of the reconstructed object, information related to its appearance is obtained, leading to better object reconstruction and improved 3D view rendering. The storage device stores relevant data generated by the system, including the reconstruction result of the target object (i.e., the rendered 3D view), the multispectral image data acquired by the multispectral imaging device, and the 3D spatial data. The display device displays the reconstructed 3D view of the target object, presenting the reconstruction results to the user.

[0078] Understandable, Figure 1 This is merely an example of a 3D reconstruction system for which an object reconstruction method provided in the embodiments of this application can be applied, and does not constitute a limitation on the embodiments of this application. The 3D reconstruction system may include more or fewer components, such as an input device.

[0079] Optionally, the multispectral imaging device can be an array multispectral camera (including an array multispectral sensor and a lens), which can acquire images of multiple spectral bands of the target scene, facilitating subsequent analysis and processing of the acquired multispectral images to obtain the material of the target object. The 3D spatial data acquisition device can be implemented in various ways. For example, the 3D spatial data acquisition device can be two traditional cameras in different poses (e.g., RGB cameras or depth images), acquiring traditional images of the target object from different perspectives, and processing these traditional images from different perspectives to obtain the visual point cloud data of the target object. Another example is a LiDAR device, which can directly obtain the 3D point cloud data of the target object by scanning it. Yet another example is a 3D spatial data acquisition device that includes a traditional camera in a different pose than the array multispectral camera. It can acquire the visual point cloud of the target object using the multispectral images acquired by the array multispectral camera, for example, by converting the multispectral image to an RGB image, which, together with the RGB image acquired by the traditional camera, forms RGB images from different perspectives, and then the visual point cloud data of the target object is obtained based on these RGB images from different perspectives. This application does not limit the specific implementation of the three-dimensional spatial data acquisition device, as long as it is a device that can directly or indirectly acquire the three-dimensional spatial data of the target object.

[0080] The object reconstruction apparatus of this application embodiment can be deployed on a terminal device. It can directly utilize the multispectral image data and three-dimensional spatial data collected by the terminal device to reconstruct a three-dimensional view of the target object, reducing reconstruction latency. The terminal device can be a smartphone, tablet computer, wearable device, vehicle terminal, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), personal digital assistant (PAD), smart screen, camera, etc. The specific type of terminal device is not limited in this application embodiment.

[0081] Optionally, the object reconstruction device can also be deployed on a cloud server to provide 3D reconstruction of the target object as a cloud service. For example, a client deployed on a terminal device can send multispectral images and 3D spatial data of the target scene to the cloud server. The object reconstruction device deployed on the cloud server uses the multispectral image data and 3D spatial data to complete the reconstruction of the target object and sends the reconstruction result to the client on the terminal device. Users can view the 3D view of the reconstructed target object through the client. In this way, the terminal device does not need to deploy the object reconstruction device, reducing the storage and computing power costs of the terminal device.

[0082] Figure 2 This is a flowchart illustrating an object reconstruction method provided in an embodiment of this application. This method can be executed by any device, equipment, platform, or cluster of devices with computing capabilities. This application does not specifically limit the specific computing device executing the method; a suitable computing device can be selected as needed. For example, it can be implemented on a terminal device, such as a smartphone or VR device. It can also be implemented on both a terminal device and a cloud device (e.g., a cloud-side server), employing an end-to-cloud collaborative architecture. Alternatively, it can be completed on a cloud device, providing 3D reconstruction services to users as a cloud service. For ease of description, the form of the executing entity will not be distinguished in the following text; all instances will be described as a 3D reconstruction system. Figure 2 As shown, the object reconstruction method provided in this application embodiment includes at least steps S201 to S204.

[0083] In step S201, a multispectral image of the target shooting scene is acquired.

[0084] Taking a mobile phone as an example, the camera module of the mobile phone has an array multispectral camera (the array multispectral camera includes an array multispectral sensor and a lens). When a user needs to perform 3D reconstruction of an object, he / she opens a software APP with 3D reconstruction function deployed on the mobile phone (the APP applies the object reconstruction method provided in the embodiments of this application). The APP calls the camera of the mobile phone. The user adjusts the camera's perspective, selects the target shooting scene, ensures that the target object to be 3D reconstructed is located in the shooting scene, triggers the shooting button, and the array multispectral sensor in the camera module generates a multispectral image of the shooting scene.

[0085] In other words, the multispectral image of the target shooting scene collected in the embodiments of this application is multispectral data with spatial resolution collected by an array multispectral sensor, indicating the spatial multispectral information of the target shooting scene, that is, describing the differences in the spatial distribution of the environmental spectrum.

[0086] It should be explained that the target object is the object that needs to be reconstructed in 3D. For example, if the object to be reconstructed is the laptop that the user is using, then the target object is the user's laptop. There can be one or more target objects, which means that one or more three-dimensional views of target objects can be reconstructed in 3D.

[0087] Figure 3 The image shows a rear view of a mobile phone.

[0088] like Figure 3 As shown, the camera module of the mobile phone 30 includes a multispectral camera 31. The mobile phone is equipped with 3D reconstruction software that provides the 3D reconstruction function according to the embodiments of this application. When a user needs to perform 3D reconstruction of a real-world object (e.g., object A), they open the 3D reconstruction software on the mobile phone, input a multispectral image of the target object through the interactive interface of the 3D reconstruction software, for example, selecting to capture a multispectral image of object A. Then, the phone's camera is invoked, the camera's shooting angle is aimed at the target shooting scene where object A is located, the shooting button is triggered, and the array of multispectral cameras acquires the multispectral signals of the target shooting scene, generating a multispectral image of the target shooting scene.

[0089] Of course, acquiring multispectral images is not limited to using the phone's own camera; it can also be achieved by receiving multispectral images transmitted from external devices. For example, a multispectral image of the target scene can be captured by a multispectral camera, and then transmitted to the phone, which then receives the multispectral image.

[0090] Figure 4 A schematic diagram of a target scene captured by a multispectral camera is shown.

[0091] like Figure 4 As shown, the multispectral camera aims its shooting angle at the target shooting scene, which consists of the target object to be reconstructed in 3D and the ambient light emitted by the LED lights as the ambient light source. The multispectral camera captures a multispectral image of the target shooting scene, which is then imported into a mobile phone. The 3D reconstruction software is then opened on the mobile phone. In the interactive interface of the 3D reconstruction software, the multispectral image of the target object is selected as the import method, and then the multispectral image is selected for import, so that the 3D reconstruction system can obtain the multispectral image.

[0092] In step S202, the three-dimensional spatial data of the target object is acquired.

[0093] Back Figure 3The camera module of the mobile phone 30 includes a multispectral camera 31, a first RGB camera 32, and a second RGB camera 33. The first RGB camera 32 and the second RGB camera 33 are positioned differently, allowing the user to acquire RGB images of the target shooting scene from different perspectives. In one example, when the user triggers the phone's shutter button, the multispectral camera 31 captures a multispectral image of the target shooting scene, while the first RGB camera 32 and the second RGB camera 33 capture first RGB images and second RGB images of the target shooting scene from different perspectives, respectively. Thus, the 3D reconstruction system obtains the three-dimensional spatial data of the target object by processing the first RGB images and the second RGB images.

[0094] Similar to multispectral image acquisition, the first RGB image and the second RGB image can also be obtained by importing them from an external source. For example, the target scene can be photographed from different angles using an RGB camera to obtain the first RGB image and the second RGB image from different perspectives (e.g., by moving a single camera around a portrait to capture images of the face at different angles and times). Then, the first RGB image and the second RGB image can be imported into a mobile phone, and the 3D reconstruction software on the phone can be opened. The RGB images from different perspectives can be obtained by selecting the import method. Then, the 3D reconstruction system can obtain the three-dimensional spatial data of the target object by processing the first RGB image and the second RGB image.

[0095] It should be noted that the three-dimensional spatial data of the target object can be expressed in various ways, such as explicit expressions like point clouds, voxels, and meshes, as well as implicit expressions like neural radiation fields. In other words, the three-dimensional spatial data can be any of the three-dimensional spatial data, such as three-dimensional point cloud data, three-dimensional voxel data, three-dimensional mesh data, or neural radiation field data. This application does not limit the three-dimensional expression method of the three-dimensional spatial data, and an appropriate expression method can be selected as needed.

[0096] In this embodiment, the first RGB camera 32 and the second RGB camera 33 can capture RGB images of the target object from different perspectives, and then the three-dimensional spatial data of the target object can be calculated using these RGB images. Alternatively, the first RGB camera 32 and the second RGB camera 33 can capture RGB images from different perspectives, including both the target object and non-target objects. For example, if the target object to be reconstructed is a computer monitor, but the images captured by the first RGB camera 32 and the second RGB camera 33 both include the computer monitor and keyboard, then the images captured by the first RGB camera 32 and the second RGB camera 33 include images of the target object monitor and non-target object keyboard. In this case, the target object to be reconstructed can be input as the monitor, and then the 3D reconstruction system can call an AI model to identify the target object monitor in the image, and then use the RGB images of the target object from different perspectives to calculate the three-dimensional spatial data of the target object monitor. In another example, the user can also select the image area of ​​the target object display to be reconstructed from the images captured by the first RGB camera 32 and the second RGB camera 33 through the interactive interface. Then, the 3D reconstruction system obtains RGB images of the target object display from different perspectives based on the image area selected by the user, and then uses them to calculate the three-dimensional spatial data of the target object display.

[0097] The following uses three-dimensional point cloud data as an example to introduce the object reconstruction method provided in the embodiments of this application.

[0098] There are several methods to obtain 3D spatial data of a target object from RGB images taken from different perspectives. For example, feature points of the target object can be extracted from each image and matched. The camera pose at each moment can be obtained using the Structure from Motion (SFM) algorithm for camera calibration. Triangulation methods can then be used to obtain the depth of the matched feature points and reconstruct a 3D point cloud, resulting in the 3D point cloud data of the target object. Another method is to acquire multiple 2D images of the target object from different perspectives using a camera. Feature points can be extracted from each 2D image to obtain a feature point set. Feature point matching can then be performed on these sets to obtain the camera pose corresponding to each 2D image. Based on the camera pose and the matched feature points, the 3D point cloud coordinates of each feature point can be reconstructed using triangulation. Yet another method is to use an AI model. After training, the AI ​​model can establish a mapping relationship between 2D images from different perspectives and 3D point clouds. For example, the first and second RGB images from different perspectives can be used as input to the AI ​​model, outputting the 3D point cloud data of the target object. In this application embodiment, the specific method for obtaining spatial data of the target object using RGB images from different perspectives is not limited. An appropriate algorithm can be selected according to the actual situation to obtain the three-dimensional point cloud data of the target object using RGB images from different perspectives.

[0099] The three-dimensional point cloud data of the target object indicates the three-dimensional spatial information of the target object, that is, the distribution information of the surface of the target object in three-dimensional space. For example, if the target object is object A, the three-dimensional spatial data of object A is expressed in the form of a three-dimensional point cloud, indicating the distribution of the outer surface of object A in three-dimensional space, such as the three-dimensional coordinates of some feature points on the outer surface of object A.

[0100] Of course, the first RGB camera 32 and the second RGB camera 33 can also be other types of traditional cameras, such as depth cameras and RYYB cameras.

[0101] It's easy to understand that when applying 2D reconstruction of a target object to obtain a 2D view of the target object, this step acquires the 2D spatial data of the target object. At this point, it's sufficient to use an RGB camera to capture an RGB image of the target scene from one viewpoint. The resulting 2D spatial data is the single-view RGB image of the target scene captured by that RGB camera. The 2D spatial data indicates the 2D spatial information of the target object. This information can be understood as the distribution of the target object's surface in 2D space. For example, by capturing an RGB image of the target scene using an RGB camera, where the target scene includes the target object A, the distribution of the surface of object A in the 2D plane can be determined from the RGB image. In other words, the shape and contour of the outer surface of object A from the shooting viewpoint can be obtained.

[0102] In step S203, the material of the target object is obtained based on the multispectral image and the three-dimensional spatial data of the target object.

[0103] After acquiring multispectral images and three-dimensional spatial data, the 3D reconstruction system processes the multispectral images and three-dimensional spatial data using a preset algorithm to obtain the material of the target object.

[0104] For example, the first step is to determine the spectral information of the ambient light in the target shooting scene. There are several methods for determining the spectrum of ambient light.

[0105] The first method is to directly measure the ambient light of the target shooting scene using a spectral measuring instrument to obtain the spectral information of the ambient light, and then use the measured spectral information of the ambient light as the ambient light parameter to configure the 3D reconstruction system.

[0106] The second method is to use ambient light with a known spectrum to illuminate the target object and create a target shooting scene. The light source parameters of the ambient light in the 3D reconstruction system are based on the known spectrum by default.

[0107] The third method involves processing multispectral images to determine the spectral information of the ambient light in the target shooting scene. For example, first, based on the multispectral image, the scene reflectance spectrum is obtained, indicating the spectral distribution of the reflectance of the target shooting scene. Then, based on the scene reflectance spectrum and k standard light sources, the ambient light spectrum is obtained, where k is a positive integer greater than 1. Specifically, the multispectral image is first inversely analyzed to obtain the scene reflectance spectrum of the actual shooting scene. Then, based on the scene reflectance spectrum, a fitting operation is performed on each of the K standard light sources to obtain K fitting coefficients. These K fitting coefficients are then used as weights to weight the spectra of the K standard light sources to obtain the ambient light spectrum. Alternatively, the multispectral image is first inversely analyzed to obtain the scene reflectance spectrum of the actual shooting scene. Then, the scene reflectance spectrum is analyzed to obtain the component proportion of each standard light source in the scene reflectance spectrum. Finally, the component proportions of each standard light source are used as weighting coefficients to weight and sum the K standard light sources to obtain the ambient light spectrum. This method eliminates the need for additional hardware for measuring ambient light sources, reducing hardware costs.

[0108] It should be noted that the K standard light sources are multiple light sources specified by the International Commission on Illumination (CIE), such as D65, D50, D55, D75, etc.

[0109] The fourth method involves using multi-view RGB images to assist multispectral image analysis to obtain a more accurate spectrum of ambient light in the target shooting scene. For example, by processing RGB image data to obtain foreground and background images, and then using these images to assist multispectral image analysis, the spectrum of ambient light in the target shooting scene can be obtained, making the estimation of the spectral information of ambient light in the target shooting scene more accurate. Specifically, the background image of the RGB image and the background image segmented from the multispectral image instance are used to estimate the spectrum of ambient light in the target shooting scene.

[0110] The light intensity recorded by the array multispectral sensor satisfies the following formula:

[0111]

[0112] Where c represents the color channel corresponding to pixel x, I c L(λ) represents the light intensity received by pixel x as recorded by the array multispectral sensor, L(λ) represents the light source spectrum, R(λ, x) represents the spectral reflectance of the material of the object corresponding to pixel x, and C represents the light intensity received by pixel x. c (λ) represents the response function of c, which is a function of the ratio of the light intensity recorded by the array multispectral sensor to the light intensity of the incident light as a function of wavelength. c (λ) is calibrated during the development of terminal equipment.

[0113] From the above formula, it can be seen that the light intensity recorded by the multispectral sensor is the integral result of the ambient light spectrum, the object's spectral reflectance, and the response function. The color values ​​of each pixel's corresponding color channel are extracted from the multispectral image, and the light intensity I of each pixel's corresponding color channel is then deduced from these values. c Thus, the spectrum of ambient light L(λ) and the response function C are known. c (λ) and the light intensity I of each pixel received by the array multispectral sensor. c The spectral reflectance of each pixel can then be calculated.

[0114] After obtaining the spectral reflectance of each pixel, the material of each pixel can be obtained by matching it with the material spectral library. Then, instance segmentation is performed on the multispectral image to obtain multiple 2D instances. Instances are generated when the multispectral image includes images of multiple target objects and a background image, such as images of object A, object B, and object C. After instance segmentation, multiple 2D instances are obtained, including 2D images of object A, object B, object C, and the background image.

[0115] The embodiments of this application do not specifically limit the specific implementation method of instance segmentation. For example, instance segmentation can be implemented based on neural networks. Specifically, a multispectral image is used as the input of a semantic segmentation model. The semantic segmentation result is input, and the multispectral image is semantically segmented according to the output semantic segmentation result to obtain multiple 2D instances.

[0116] It should be explained that the material spectral library mentioned in the embodiments of this application stores records a variety of materials and the spectral reflectance of each material. The material spectral library can exist in the form of a database, recording and storing a variety of materials and the spectral reflectance of each material. The material spectral library can also exist in the form of a mapping table, recording and maintaining the mapping relationship between each material and its reflectance. The embodiments of this application do not specifically limit the implementation form of the material spectral library, as long as the material corresponding to the reflectance can be found in it.

[0117] The material of each 2D instance is determined based on the material of its pixels. Optionally, a multispectral image can be used as input to the image segmentation model, outputting instance segmentation results to obtain multiple 2D image instances.

[0118] In another example, to more accurately obtain the material of each 2D instance in a multispectral image, the spectral reflectance of each 2D instance can be obtained based on its reflection spectrum, ambient light spectrum, and the spectrum of the target standard light source. The ambient light spectrum refers to the spectrum of the ambient light in the target shooting scene. Based on the spectral reflectance of each 2D instance, the material of each 2D instance is retrieved from a material spectral library. The material spectral library records multiple materials and the corresponding spectral reflectance of each material under the target standard light source.

[0119] For example, the reflectance spectrum of each 2D instance is divided by the spectrum of ambient light and then multiplied by the spectrum of the target standard light source to obtain the spectral reflectance of each 2D instance under the standard light source. Then, the material of each 2D instance under the target standard light source is obtained by searching the material spectrum library. In this way, the material of each 2D instance can be accurately obtained.

[0120] Thus, multispectral material analysis results were obtained by processing multispectral images. The multispectral material analysis results include the material corresponding to each 2D instance of the multispectral image.

[0121] Figure 5 A schematic diagram illustrating the multispectral image data processing procedure according to an embodiment of this application is shown. Figure 5As shown, by processing the multispectral image in two branches, the ambient light spectrum and multispectral material analysis results are obtained, respectively. For example, one branch of the multispectral image processing involves estimating the ambient light of the target shooting scene using the multispectral image to obtain the ambient light spectrum. For instance, the multispectral image is first inversely processed to obtain the scene reflection spectrum of the actual shooting scene. Then, based on the scene reflection spectrum, a fitting operation is performed on K standard light sources to obtain K fitting coefficients. These K fitting coefficients are then used as weights to weight the spectra of the K standard light sources to obtain the ambient light spectrum. The other branch of processing involves processing the multispectral image to obtain the material corresponding to each 2D instance in the multispectral image. For example, first, image instance segmentation is performed on the multispectral image to obtain multiple 2D instances. Then, the average reflectance spectrum of each 2D instance is obtained by averaging the reflectance spectrum of each 2D instance. The average reflectance spectrum of each 2D instance is used as the reflectance spectrum of each 2D instance. Then, based on the reflectance spectrum of each 2D instance, the spectrum of ambient light, and the spectrum of the target standard light source (e.g., D65), the spectral reflectance of each 2D instance is obtained. The spectrum of ambient light is the spectrum of the ambient light of the target shooting scene. Based on the spectral reflectance of each 2D instance, the material of each 2D instance is retrieved from the material spectral library. The material spectral library records multiple materials and the corresponding spectral reflectance of each material under the target standard light source.

[0122] Then, the 3D spatial data is processed to obtain 3D instance information. For example, instance segmentation of 3D point cloud data can be performed to obtain 3D instances of each target object. Instance segmentation of 3D point clouds can be achieved through various methods to obtain 3D instances of each target object. For example, the cluster-free panoptic segmentation of 3D LiDAR point clouds (CPSeg) method can be used to segment 3D point clouds to obtain 3D instances of each target object. Another example is instance segmentation of 3D point cloud data using machine learning. For instance, 3D point cloud data can be used as input to a 3D point cloud segmentation model, and the segmentation result of the 3D point cloud can be output, thus obtaining the 3D instance corresponding to each target object. For example, if the target shooting scene includes target objects A, B, and C, then after 3D instance segmentation of the 3D point cloud data, 3D instances of object A, object B, and object C can be obtained. This application does not limit the instance segmentation method for 3D point clouds; any feasible instance segmentation method for 3D point clouds can be selected as needed.

[0123] By combining the 3D instance and multispectral material analysis results, the 3D instance material analysis results are obtained, which include the materials corresponding to each 3D instance. Figure 6This diagram illustrates a process for processing 3D instance material analysis results by combining 3D instance data and multispectral material analysis results. (For example...) Figure 6 As shown, a registration operation is performed on each 2D instance and each 3D instance to match the spatial pixels of each 2D instance to the 3D instance corresponding to each target object; based on the material corresponding to the spatial pixels of the 3D instance corresponding to each target object, the material of each target object is obtained.

[0124] For example, image registration is performed using calibration parameters of a multispectral sensor and an RGB camera (e.g., the pose of the multispectral sensor and the pose of the RGB camera). This matches the spatial pixels of each 2D instance to the corresponding 3D instance of each target object. Each pixel in the 2D instance carries a material information. For the material of the matched pixels on any 3D instance, a voting system is used to determine the final material of the 3D instance. For instance, if the ratio of iron, aluminum, and copper in the matched pixels on a 3D instance is 3:2:1, the material of that 3D instance is determined to be iron because the most matched pixels are iron. This method accurately obtains the material of each 3D instance, which in turn obtains the material of each target object.

[0125] In step S204, the 3D reconstruction result of the target object is obtained based on the three-dimensional spatial data and material of the target object.

[0126] Through the above steps, the 3D reconstruction system processes multispectral images and 3D spatial data to obtain information about each 3D instance, its material, and the ambient light source. Finally, based on each 3D instance, its material, and the spectrum of the ambient light, the system renders each 3D instance to obtain a 3D view of each target object.

[0127] By introducing materials into 3D instance rendering, the appearance attributes of the target object corresponding to the material are obtained using the material, such as the gloss and reflectivity of the target object. The reflectivity is used to obtain the reflection spectrum of the target object under ambient light. Then, the gloss and reflection spectrum are used to render the 3D instance, thereby improving the 3D rendering effect.

[0128] For example, by obtaining the gloss of each 3D instance through its material, the 3D instance is rendered based on its gloss, making the rendered 3D view more textured. For instance, if the material of a 3D instance is copper, then the 3D instance is rendered with a copper gloss.

[0129] For example, by obtaining the reflectivity of each 3D instance through its material, and then calculating the reflection spectrum of the 3D instance under ambient light based on the reflectivity and the spectrum of ambient light, the 3D instance is rendered, making the color of the rendered 3D view more accurate.

[0130] In one example, the rendering of a target object can be accomplished using a 3D Gaussian splatting (3D GS) model. For instance, 3D point cloud data is used as input to a 3D Gaussian model, and the output is a corresponding 3D Gaussian point cloud. Each point in the 3D Gaussian point cloud includes material information, and the color representation parameters of the 3D Gaussian model are determined based on the material of each target object. Based on the 3D Gaussian point cloud and the spectrum of ambient light, the target object is rendered to obtain a 3D view of the target object. Processing the 3D point cloud data using a 3D Gaussian model and then using the processed 3D Gaussian point cloud for rendering further improves the 3D rendering effect.

[0131] Specifically, the color representation parameters of the 3D Gaussian sphere are initialized as the product of the material reflection spectrum under a standard light source matched by the material analysis results and a set of spherical harmonic functions. The opacity and 3D covariance matrix (representing the scaling degree) of the 3D Gaussian sphere are randomly initialized. The 3D GS point cloud is used to learn the representation using the standard 3D GS learning process to obtain the final 3D GS representation. For each instance material in the 3D GS instance material analysis results, combined with ambient light information and the default viewpoint, the rendering results of each target pixel are reconstructed based on the 3D GS point cloud using methods such as ray tracing, ultimately obtaining the rendered 3D view of the target object.

[0132] It should be noted that this application supports the reconstruction of a 3D view of a single target object at a time, as well as the reconstruction of 3D views of multiple target objects at a time. When a user wants to reconstruct the 3D view of a single target object at a time, the shooting angle is adjusted during the shooting process so that the shooting scene includes a target object. Then, using the acquired multispectral images and three-dimensional spatial data, the 3D reconstruction of the target object can be achieved, resulting in a 3D view of the target object with a better reconstruction effect (see the description above for details).

[0133] When a user wants to reconstruct 3D views of multiple target objects at once, the shooting angle is adjusted during shooting so that the shooting scene includes multiple target objects (e.g., target objects A, B, and C). Multispectral images and 3D spatial data of the shooting scene are acquired. Then, according to the algorithm steps described above in the embodiments of this application, the material of each target object is calculated by combining the multispectral images and 3D spatial data. For example, spectral analysis is performed on the acquired multispectral images to obtain the material of each pixel. Then, instance segmentation is performed on the multispectral images to obtain multiple 2D instances. The material of each 2D instance is obtained based on the pixel material of each 2D instance. Then, instance segmentation is performed on the 3D spatial data to obtain multiple 3D instances. Then, a registration operation is performed to match the spatial pixels of each 2D instance to the 3D instances of each target object (registration will automatically match 2D instances and 3D instances of the same target object), thereby obtaining the material of the 3D instances of each target object. Finally, the 3D views of each target object are reconstructed using the materials of the 3D instances of each target object.

[0134] The 3D view of the target object reconstructed by the 3D reconstruction system of this application embodiment also supports editing of the 3D view. In addition, this application embodiment also provides an object editing method, which can efficiently and flexibly edit the 3D view of the target object reconstructed by the 3D reconstruction system of this application embodiment.

[0135] Figure 7 This illustration shows a flowchart of an object editing method provided in an embodiment of this application. This method can be executed by any device, equipment, platform, or cluster of devices with computing capabilities. This application does not specifically limit the specific computing device executing the method; a suitable computing device can be selected as needed. For example, it can be implemented on a terminal device, such as a smartphone or VR device. It can also be implemented on both a terminal device and a cloud device (e.g., a cloud-side server), employing an end-to-cloud collaborative architecture. Alternatively, it can be completed on a cloud device, providing 3D editing services to users as a cloud service. For ease of description, the form of the executing entity will not be distinguished in the following text; all instances will be described as a 3D editing system. Figure 7 As shown, the object editing method provided in this application embodiment includes at least steps S701 to S702.

[0136] In step S701, a three-dimensional view of the target object is obtained.

[0137] The three-dimensional view of the target object is reconstructed based on the object reconstruction method provided in the embodiments of this application. For the specific reconstruction method, please refer to the description above. For the sake of brevity, it will not be repeated here.

[0138] Once the 3D view is reconstructed by the 3D reconstruction system, it can be edited again. In other words, the APP with 3D reconstruction function mentioned above integrates the 3D editing system of this application embodiment. After the 3D view of the target object is reconstructed by the 3D reconstruction method provided by this application embodiment, the 3D view can also be edited, so that the 3D view can be edited and modified according to the user's needs, or when the user finds errors in the reconstructed 3D view, it can be edited and modified.

[0139] Of course, the 3D editing system provided in this application can also edit other 3D views, and is not limited to editing the 3D views reconstructed by the 3D reconstruction system in the embodiments of this application.

[0140] In step S702, the 3D view is edited in response to the editing parameters input by the user.

[0141] Users can input editing parameters through the editing interface of an app with 3D reconstruction capabilities. The 3D editing system will then respond to these parameters to edit and update the 3D view.

[0142] Editing parameters include one or more of the following: ambient light editing parameters, 3D instance editing parameters, and viewpoint editing parameters. Among them, ambient light editing parameters include ambient light parameters, ambient light maps, or ambient light model parameters, and ambient light parameters include one or more of the following: ambient light spectral parameters, illumination angle parameters, and illumination position parameters. 3D instance editing parameters include one or more of the following: 3D instance position parameters, 3D instance size parameters, 3D instance material parameters, 3D instance modification parameters, 3D instance addition parameters, and 3D instance deletion parameters. Viewpoint editing parameters include one or more of the following: viewpoint adjustment parameters, view zoom-out parameters, and view zoom-in parameters.

[0143] For example, an app with 3D reconstruction capabilities has an editing interface. Users can input ambient light parameters through this interface. There are multiple ways to input ambient light parameters, such as directly inputting ambient light parameters, importing ambient light textures, or importing ambient light models. The 3D editing system responds to the ambient light editing information, modifies the ambient light to generate new ambient light, and then re-renders the 3D view based on the new ambient light.

[0144] Users can also edit 3D instances through the interactive interface, such as replacing, copying, deleting, dragging, scaling, and modifying materials. For example, if a user finds that the material of the reconstructed 3D view is incorrect—it should be metal but is being reconstructed as plastic—they can edit the material to change it to metal, and then re-render the 3D view based on the edited metal material to obtain a 3D view with the metal material.

[0145] Users can also edit the viewpoint and size of the 3D view through the editing interface. For example, they can adjust the viewpoint or zoom the 3D view through the editing interface. The 3D editing system responds to the editing operation by adjusting the viewpoint and zooming the 3D view.

[0146] Based on the same concept as the aforementioned embodiment of the object reconstruction method, this application also provides an object reconstruction apparatus 800, which can be deployed in a terminal device or a cloud server to improve the reconstruction effect of the target object. The object reconstruction apparatus 800 includes components for implementing... Figure 2-6 The units or modules of each step in the object reconstruction method shown.

[0147] Figure 8 This is a schematic diagram of the structure of an object reconstruction device provided in an embodiment of this application. Figure 8 As shown, the object reconstruction device 800 includes at least a first acquisition module 801, a second acquisition module 802, a determination module 803, and a reconstruction module 804. The first acquisition module 801 acquires a multispectral image of a target shooting scene, which includes a target object. The multispectral image indicates the multispectral information of the target shooting scene's space. The second acquisition module 802 acquires spatial data of the target object, which indicates the target object's two-dimensional or three-dimensional spatial information. The determination module 803 determines the material of the target object based on the multispectral image and the target object's spatial data. The reconstruction module 804 obtains a reconstruction result of the target object based on the target object's spatial data and the target object's material. The reconstruction result includes a two-dimensional or three-dimensional view of the target object.

[0148] In one possible implementation, the spatial data of the target object includes the three-dimensional point cloud data of the target object; the first determining module 803 is specifically used to: perform instance segmentation on the multispectral image to obtain at least one 2D instance; obtain the material of each 2D instance based on the reflectance spectrum of each 2D instance in the at least one 2D instance; perform instance segmentation on the three-dimensional point cloud data to obtain a 3D instance of the target object; and obtain the material of the target object based on the material of each 2D instance and the 3D instance of the target object.

[0149] In another possible implementation, a specific implementation of obtaining the material of each 2D instance based on the reflection spectrum of each 2D instance in at least one 2D instance is as follows: Based on the reflection spectrum of each 2D instance, the spectrum of ambient light, and the spectrum of the target standard light source, the spectral reflectance of each 2D instance is obtained, where the spectrum of ambient light is the spectrum of the ambient light of the target shooting scene; Based on the spectral reflectance of each 2D instance, the material of each 2D instance is obtained by retrieving it from a material spectral library, where the material spectral library records multiple materials and the spectral reflectance of each material under the target standard light source.

[0150] In another possible implementation, the object reconstruction apparatus 800 provided in this application further includes a second determining module 805, which is used to obtain a scene reflection spectrum based on a multispectral image, the scene reflection spectrum indicating the reflection spectrum distribution of the target shooting scene; and to obtain the spectrum of ambient light based on the scene reflection spectrum and k standard light sources, where k is a positive integer greater than 1.

[0151] In another possible implementation, the second determining module 805 is specifically used to: perform fitting operations on K standard light sources based on the scene reflection spectrum to obtain K fitting coefficients; and use the K fitting coefficients as weights to perform weighting operations on the spectra of the K standard light sources to obtain the spectrum of ambient light.

[0152] In another possible implementation, the spectrum of ambient light is obtained based on spectral measurements of the ambient light in the scene being photographed.

[0153] In another possible implementation, the 3D point cloud data of the target object is obtained based on RGB images from multiple perspectives; the spectrum of ambient light is obtained based on multispectral images and RGB images from multiple perspectives.

[0154] In another possible implementation, the first determining module 803 is specifically used to: perform a registration operation on each 2D instance and the 3D instance of the target object, matching the spatial pixels of each 2D instance to the 3D instance of the target object; and obtain the material of the target object based on the material corresponding to the spatial pixels matched to the 3D instance of the target object.

[0155] In another possible implementation, the reconstruction module 804 is specifically used to: render the target object based on the 3D instance of the target object, the material of the target object, and the spectrum of ambient light to obtain a 3D view of the target object.

[0156] In another possible implementation, a specific approach to obtaining the reconstruction result of the target object based on its spatial data and material is as follows: 3D point cloud data is used as input to a 3D Gaussian model, and the output is a 3D Gaussian point cloud corresponding to the 3D point cloud data. Each point in the 3D Gaussian point cloud includes material information, and the color representation parameters of the 3D Gaussian model are determined based on the material of the target object. Based on the 3D Gaussian point cloud and the spectrum of ambient light, the target object is rendered to obtain a 3D view of the target object.

[0157] In another possible implementation, the second acquisition module 802 is specifically used to: acquire images of the target shooting scene from multiple different perspectives; and obtain three-dimensional point cloud data of the target object based on the images from multiple different perspectives.

[0158] In another possible implementation, the images from multiple different viewpoints include multiple RGB images from multiple different viewpoints; or, the images from multiple different viewpoints include depth images and RGB images from multiple different viewpoints; or, the images from multiple different viewpoints include multiple RGB images from multiple different viewpoints, wherein a portion of the RGB images from the multiple RGB images from multiple different viewpoints is obtained based on multispectral image conversion.

[0159] The object reconstruction apparatus 800 according to the embodiments of this application can correspond to the execution of the methods described in the embodiments of this application, and the above and other operations and / or functions of each module in the object reconstruction apparatus 800 are respectively for implementing Figure 2-6 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0160] Based on the same concept as the aforementioned embodiment of an object editing method, this application also provides an object editing device 900, which can be deployed in a terminal device or a cloud server to flexibly edit the reconstruction results. The object editing device 900 includes components for implementing... Figure 9 The units or modules of each step in the object editing method shown.

[0161] Figure 9 This is a schematic diagram of the structure of an object editing device provided in an embodiment of this application. Figure 9 As shown, the object editing device 900 includes at least a third acquisition module 901 and an editing module 902. The third acquisition module 901 is used to acquire a two-dimensional view or a three-dimensional view of a target object, which is reconstructed based on the object reconstruction method provided in the embodiments of this application. The editing module 902 is used to edit the two-dimensional view or the three-dimensional view in response to the editing parameters input by the user.

[0162] In one possible implementation, the editing parameters include one or more of the following: ambient light editing parameters, 3D instance editing parameters, and viewpoint editing parameters; wherein, the ambient light editing parameters include ambient light parameters, ambient light maps, or ambient light model parameters, and the ambient light parameters include one or more of the following: ambient light spectral parameters, illumination angle parameters, and illumination position parameters; the 3D instance editing parameters include one or more of the following: 3D instance position parameters, 3D instance size parameters, 3D instance material parameters, 3D instance modification parameters, 3D instance addition parameters, and 3D instance deletion parameters; and the viewpoint editing parameters include one or more of the following: viewpoint adjustment parameters, view zoom-out parameters, and view zoom-in parameters.

[0163] The object editing apparatus 900 according to the embodiments of this application can correspond to the execution of the methods described in the embodiments of this application, and the above and other operations and / or functions of each module in the object editing apparatus 900 are respectively for implementing Figure 9 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0164] The object reconstruction device and object editing device provided in this application embodiment can be deployed in one terminal device or in different terminal devices to achieve distributed deployment. Alternatively, some modules of the object reconstruction device can be deployed in one computing device, while others can be deployed in the same terminal device as the object editing device.

[0165] Figure 10 A schematic diagram illustrating one deployment method of the object reconstruction apparatus and object editing apparatus of this application is shown. Figure 10 As shown. The first acquisition module 801, the second acquisition module 802, and the determination module 803 in the object reconstruction device are deployed in terminal device 1, used to process multispectral images and three-dimensional spatial data to obtain the material of the 3D instance. The reconstruction module 804 in the object reconstruction device, and the third acquisition module 901 and editing module 902 in the object editing device are both deployed in terminal device 2. In terminal device 2, the rendering of the material-based 3D instance to obtain a 3D view, and the editing of the 3D view, are implemented. Terminal device 2 is externally connected to a display device and an input device. The display device, for example, can be a monitor, used to display the 3D view for user viewing. The input device, for example, can be a mouse and / or keyboard and / or touchscreen, used to input editing parameters to terminal device 2 for editing the 3D view.

[0166] certainly Figure 10This is merely an example of a distributed deployment and does not constitute a limitation on the embodiments of this application. Other deployment methods may also be used in other examples, such as deploying the first acquisition module 801 and the second acquisition module 802 on one terminal device, and deploying the determination module 803, the reconstruction module 804, the third acquisition module 901 and the editing module 902 on another terminal device.

[0167] This application embodiment also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figure 2-7 The method described.

[0168] Figure 11 A schematic diagram of the structure of a computing device provided in an embodiment of this application.

[0169] like Figure 11 As shown, the computing device 1100 includes at least one processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1103 is used to send and / or receive data from other devices. The memory 1102 stores computer instructions, which the processor 1101 executes to perform the methods described in the foregoing method embodiments, thereby achieving two-dimensional or three-dimensional reconstruction of the target object, improving the reconstruction effect, and allowing for flexible editing of the reconstruction results.

[0170] It should be understood that, in the embodiments of this application, the processor 1101 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0171] The memory 1102 may include read-only memory and random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include non-volatile random access memory.

[0172] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0173] It should be understood that the computing device 1100 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figure 2-7 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0174] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0175] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0176] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0177] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0179] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An object reconstruction method, characterized in that, include: Acquire a multispectral image of a shooting scene, wherein the shooting scene includes a target object, and the multispectral image indicates the spatial multispectral information of the shooting scene; Acquire spatial data of the target object, wherein the spatial data indicates two-dimensional or three-dimensional spatial information of the target object; Based on the multispectral image and the spatial data of the target object, the material of the target object is obtained; Based on the spatial data and material of the target object, a reconstruction result of the target object is obtained, which includes a two-dimensional view or a three-dimensional view of the target object.

2. The method according to claim 1, characterized in that, The spatial data of the target object includes the three-dimensional representation data of the target object; The process of obtaining the material of the target object based on the multispectral image and the spatial data of the target object includes: The multispectral image is segmented to obtain at least one 2D instance; The material of each 2D instance is obtained based on the reflection spectrum of each 2D instance in the at least one 2D instance; Instance segmentation is performed on the three-dimensional representation data to obtain a 3D instance of the target object; The material of the target object is obtained based on the material of each 2D instance and the 3D instance of the target object.

3. The method according to claim 2, characterized in that, The process of obtaining the material of each 2D instance based on the reflection spectrum of each 2D instance in the at least one 2D instance includes: Based on the reflectance spectrum and ambient light spectrum of each 2D instance, the spectral reflectance of each 2D instance is obtained, wherein the ambient light spectrum is the ambient light spectrum of the shooting scene. The material of each 2D instance is obtained by retrieving its spectral reflectance from the material spectral library. The material spectral library records a variety of materials and the spectral reflectance of each of the various materials.

4. The method according to claim 2 or 3, characterized in that, Also includes: Based on the multispectral image, a scene reflectance spectrum is obtained, which indicates the reflectance spectral distribution of the captured scene; Based on the scene reflection spectrum and k standard light sources, the spectrum of the ambient light is obtained, where k is a positive integer greater than 1.

5. The method according to claim 4, characterized in that, The process of obtaining the ambient light spectrum based on the scene's reflectance spectrum and K standard light sources includes: Based on the scene reflection spectrum, a fitting operation is performed on the K standard light sources to obtain K fitting coefficients; The K fitting coefficients are used as weights to weight the spectra of the K standard light sources to obtain the spectrum of the ambient light.

6. The method according to claim 3, characterized in that, The spectrum of the ambient light is obtained based on the spectral measurement of the ambient light in the shooting scene.

7. The method according to claim 3, characterized in that, The three-dimensional representation data of the target object is obtained based on RGB images from multiple perspectives; The spectrum of the ambient light is obtained based on the multispectral image and the RGB images from the multiple viewpoints.

8. The method according to any one of claims 2-7, characterized in that, The process of obtaining the material of the target object based on the materials of each 2D instance and the 3D instance of the target object includes: A registration operation is performed on each 2D instance and the 3D instance of the target object to match the spatial pixels of each 2D instance to the 3D instance of the target object. The material of the target object is obtained based on the material corresponding to the spatial pixel points matched by the 3D instance of the target object.

9. The method according to any one of claims 3-8, characterized in that, The process of obtaining the reconstruction result of the target object based on the spatial data and material of the target object includes: Based on the 3D instance of the target object, the material of the target object, and the spectrum of the ambient light, the target object is rendered to obtain a 3D view of the target object.

10. The method according to any one of claims 3-8, characterized in that, The process of obtaining the reconstruction result of the target object based on the spatial data and material of the target object includes: The three-dimensional representation data is used as input to a 3D Gaussian model, and the 3D Gaussian point cloud corresponding to the three-dimensional representation data is output. Each point in the 3D Gaussian point cloud includes material information, and the color representation parameters of the 3D Gaussian model are determined based on the material of the target object. Based on the 3D Gaussian point cloud and the spectrum of the ambient light, the target object is rendered to obtain a three-dimensional view of the target object.

11. The method according to any one of claims 1-10, characterized in that, The acquisition of the spatial data of the target object includes: Acquire images of the shooting scene from multiple different perspectives; Based on the images from multiple different perspectives, three-dimensional representation data of the target object are obtained.

12. The method according to claim 11, characterized in that, The images from multiple different perspectives include multiple RGB images from different perspectives; Alternatively, the multiple images from different perspectives may include depth images and RGB images from different perspectives; Alternatively, the multiple images from different perspectives may include multiple RGB images from different perspectives, wherein some of the RGB images from the multiple RGB images from different perspectives are obtained based on the multispectral image conversion.

13. An object editing method, characterized in that, include: Obtain a two-dimensional or three-dimensional view of the target object, wherein the two-dimensional or three-dimensional view of the target object is reconstructed based on the object reconstruction method as described in any one of claims 1-12; The two-dimensional or three-dimensional view is edited in response to user-inputted editing parameters.

14. The method according to claim 13, characterized in that, The editing parameters include one or more of the following: ambient light editing parameters, 3D instance editing parameters, and viewpoint editing parameters; The ambient light editing parameters include ambient light parameters, ambient light maps, or ambient light model parameters. The ambient light parameters include one or more of the ambient light spectral parameters, illumination angle parameters, and illumination position parameters. The 3D instance editing parameters include one or more of the following: 3D instance position parameters, 3D instance size parameters, 3D instance material parameters, 3D instance modification parameters, 3D instance addition parameters, and 3D instance deletion parameters. The view editing parameters include one or more of the following: view adjustment parameters, view zoom-out parameters, and view zoom-in parameters.

15. An object reconstruction apparatus, characterized in that, include: The first acquisition module is used to acquire a multispectral image of a shooting scene, wherein the shooting scene includes a target object, and the multispectral image indicates the spatial multispectral information of the shooting scene. The second acquisition module is used to acquire spatial data of the target object, wherein the spatial data indicates two-dimensional or three-dimensional spatial information of the target object. The first determining module is used to determine the material of the target object based on the multispectral image and the spatial data of the target object; The reconstruction module is used to obtain the reconstruction result of the target object based on the spatial data and material of the target object. The reconstruction result includes a two-dimensional view or a three-dimensional view of the target object.

16. An object editing device, characterized in that, include: The third acquisition module acquires a two-dimensional or three-dimensional view of the target object, wherein the two-dimensional or three-dimensional view of the target object is reconstructed based on the object reconstruction method as described in any one of claims 1-12; The editing module is used to edit the two-dimensional or three-dimensional view in response to user-inputted editing parameters.

17. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-14 to be implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-14 to be implemented.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the method as described in any one of claims 1-14 to be implemented.

Images