Three-dimensional modeling method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a three-dimensional modeling method and device, electronic equipment and a readable storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining shot images containing a to-be-reconstructed object under a plurality of visual angles, determining the camera shooting parameters of each shot image, and carrying out the point cloud reconstruction according to each shot image, obtaining sparse point cloud data, wherein the camera shooting parameters comprise camera poses; segmenting a to-be-reconstructed object in each shot image, and determining an object mask of the to-be-reconstructed object in each shot image; and performing three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the camera shooting parameters, the shot images and the object masks to obtain a three-dimensional model of the object to be reconstructed. The technical problem that the data redundancy of the reconstructed three-dimensional model is high is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional modeling method, device, electronic device and readable storage medium. Background Art

[0002] 3D modeling technology can extract geometric and appearance information from multi-view photos of the same scene using computer vision algorithms, and then reconstruct a 3D model of the scene. Current 3D modeling technologies all fully reconstruct the scene captured in the photo, which often includes background information or irrelevant objects that the user does not care about, resulting in the data of the reconstructed 3D model being too redundant.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a three-dimensional modeling method, device, electronic device and readable storage medium, aiming to solve the technical problem of high data redundancy of the reconstructed three-dimensional model.

[0005] To achieve the above objectives, the present application provides a three-dimensional modeling method, comprising:

[0006] Acquire captured images containing the object to be reconstructed from multiple perspectives, determine the camera shooting parameters of each captured image, and perform point cloud reconstruction based on each captured image to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture;

[0007] Segmenting the objects to be reconstructed in each captured image respectively to obtain reconstructed images of the objects to be reconstructed corresponding to each captured image;

[0008] According to the sparse point cloud data, the shooting parameters of each camera and each reconstructed image, a three-dimensional reconstruction is performed on the object to be reconstructed to obtain a three-dimensional model of the object to be reconstructed.

[0009] In one embodiment, the steps of respectively determining the camera shooting parameters of each captured image, and reconstructing the point cloud according to each captured image to obtain sparse point cloud data include:

[0010] Calling a preset feature extraction model to determine similar feature points between at least two captured images;

[0011] Extract target feature points that match similar feature points from each captured image;

[0012] Determine the camera shooting parameters corresponding to each of the captured images based on the coordinates of the target feature points corresponding to each of the captured images;

[0013] The target feature points corresponding to each captured image are mapped to three-dimensional space to obtain sparse point cloud data.

[0014] In one embodiment, the reconstructed image includes a target reconstructed image; each captured image includes a target captured image and other captured images, and the steps of segmenting the object to be reconstructed in each captured image to obtain the reconstructed image of the object to be reconstructed corresponding to each captured image include:

[0015] receiving a click instruction from a user, determining a target mask region of the object to be reconstructed in the target captured image from the click instruction, and determining a target reconstructed image of the object to be reconstructed in the target captured image based on the target mask region;

[0016] According to the target mask area, each other captured image is segmented to obtain a reconstructed mask area of ​​each other captured image;

[0017] Based on the reconstruction mask regions of the other captured images, reconstructed images of the objects to be reconstructed corresponding to the other captured images are determined.

[0018] In one embodiment, the step of performing three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the shooting parameters of each camera, and each reconstructed image to obtain the three-dimensional model of the object to be reconstructed includes:

[0019] Perform Gaussian initialization on the sparse point cloud data to obtain an initial three-dimensional Gaussian;

[0020] According to the shooting parameters of each camera, the initial three-dimensional Gaussian is projected onto a two-dimensional plane to obtain the renderings corresponding to the initial three-dimensional Gaussian under the shooting parameters of each camera;

[0021] The loss value between each rendering image and the corresponding reconstructed image is calculated successively, and the three-dimensional Gaussian parameters of the initial three-dimensional Gaussian are updated successively according to the loss value, so as to update the initial three-dimensional Gaussian successively to obtain an updated three-dimensional Gaussian;

[0022] Based on the updated three-dimensional Gaussian, a three-dimensional model of the object to be reconstructed is determined, wherein the loss value is determined based on the pixel-by-pixel loss, structural similarity loss and opacity loss between the rendering and the corresponding reconstructed image, and the three-dimensional Gaussian parameters include opacity parameters.

[0023] In one embodiment, the step of determining the three-dimensional model of the object to be reconstructed based on the updated three-dimensional Gaussian includes:

[0024] The target points whose opacity in the updated three-dimensional Gaussian is less than a preset opacity threshold are eliminated to obtain a three-dimensional model of the object to be reconstructed.

[0025] In one embodiment, after performing three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the shooting parameters of each camera and each reconstructed image to obtain a three-dimensional model of the object to be reconstructed, the method further includes:

[0026] Projecting the reconstructed points in the 3D model to a world point cloud based on a world coordinate system to obtain a world point cloud including the reconstructed point cloud, and filtering out scene point clouds that do not match the reconstructed point cloud in the world point cloud;

[0027] Determine the target bottom surface in the scene point cloud, and generate a Gaussian dense point cloud bounding box of the object to be reconstructed in the world point cloud based on the normal vector of the target bottom surface and the target bottom surface, and determine the bottom center coordinates of the target bottom surface;

[0028] Determine the translation vector according to the relative position between the coordinates of the center of the bottom surface and the coordinates of the origin of the world coordinate system;

[0029] According to the preset rotation matrix and translation vector, the coordinates of the Gaussian dense point cloud bounding box and the points within the Gaussian dense point cloud bounding box are transformed to obtain the world 3D model of the object to be reconstructed in the world coordinate system.

[0030] In one embodiment, the step of determining the bottom surface of the target in the scene point cloud includes:

[0031] Based on the preset random sampling algorithm, plane estimation is performed in the scene point cloud to obtain multiple candidate bottom surfaces;

[0032] Determine a target bottom surface that meets a preset bottom surface condition among the candidate bottom surfaces;

[0033] The preset bottom surface conditions include: the average distance between each preset reconstruction point in the reconstructed point cloud and the candidate bottom surface is less than the preset point cloud span threshold, the difference between the number of points in the point clouds on both sides of the candidate bottom surface is less than the preset first point cloud threshold, and the number of points in the candidate bottom surface is greater than the preset second point cloud threshold.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a three-dimensional modeling device, the device comprising:

[0035] An acquisition module is used to acquire photographed images containing the object to be reconstructed from multiple perspectives, and respectively determine the camera shooting parameters of each photographed image, and perform point cloud reconstruction according to each photographed image to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture;

[0036] A segmentation module is used to segment the objects to be reconstructed in each captured image to obtain a reconstructed image of the object to be reconstructed corresponding to each captured image;

[0037] The reconstruction module is used to perform three-dimensional reconstruction of the object to be reconstructed based on the sparse point cloud data, the shooting parameters of each camera and each reconstructed image to obtain a three-dimensional model of the object to be reconstructed.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: a memory, a processor, and a program of a three-dimensional modeling method stored in the memory and executable on the processor. When the program of the three-dimensional modeling method is executed by the processor, the steps of the three-dimensional modeling method as described above can be implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which is stored a program for implementing the three-dimensional modeling method. When the program of the three-dimensional modeling method is executed by a processor, the steps of the three-dimensional modeling method as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned three-dimensional modeling method when executed by a processor.

[0041] One or more technical solutions proposed in the present application have at least the following technical effects: acquiring captured images containing objects to be reconstructed from multiple perspectives, and determining the camera shooting parameters of each captured image, and performing point cloud reconstruction based on each captured image to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture; segmenting the objects to be reconstructed in each captured image to obtain reconstructed images of the objects to be reconstructed corresponding to each captured image; performing three-dimensional reconstruction of the objects to be reconstructed based on the sparse point cloud data, the camera shooting parameters and the reconstructed images to obtain a three-dimensional model of the objects to be reconstructed.

[0042] Since the sparse point cloud data can be represented as a three-dimensional framework reflected by each captured image, and the camera shooting posture in the camera shooting parameters of each captured image can reflect the position of the camera when shooting the object to be reconstructed at different viewing angles, and since the reconstructed image is obtained based on the segmentation of the object to be reconstructed in each captured image, each reconstructed image can individually reflect the object to be reconstructed at different viewing angles, that is, the reconstructed image represents the object to be reconstructed, but not the scene where the object to be reconstructed is located. Furthermore, through the camera shooting parameters, sparse point cloud data, and each reconstructed image, the object to be reconstructed at different viewing angles can be integrated into the sparse point cloud data in the three-dimensional space to obtain the three-dimensional model of the object to be reconstructed, so that the three-dimensional model represents the representation of the object to be reconstructed in the three-dimensional space, but does not represent the scene where the object to be reconstructed is located (for example, the background and\or irrelevant objects, etc.), thereby reducing the data redundancy of the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A schematic diagram of a flow chart of an embodiment of a three-dimensional modeling method of the present application;

[0046] Figure 2 A schematic diagram of the process of determining camera shooting parameters and sparse point cloud data in the three-dimensional modeling method of this application;

[0047] Figure 3 A schematic diagram of an example of a three-dimensional modeling method in the present application;

[0048] Figure 4 A flowchart of another example of the three-dimensional modeling method of the present application;

[0049] Figure 5 This is a schematic diagram of the structure of a three-dimensional modeling device according to an embodiment of the present application;

[0050] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the three-dimensional modeling method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] 3D reconstruction can be achieved using technologies such as neural radiation field Nerf and 3D Gaussian representation. Currently, 3D modeling based on Nerf and 3D Gaussian generally models the entire shooting scene, which often includes background information and / or irrelevant objects that users do not care about. This not only causes the emergence of invalid data, but also greatly limits the application of 3D reconstruction in product promotion, video implantation, etc. The objects to be reconstructed are too strongly coupled with the scene and are difficult to separate.

[0055] In order to solve the above defects, the embodiment of the present application provides a three-dimensional modeling method. By acquiring the captured images under multiple perspectives input by the user, the user can select the object to be reconstructed in any of the input captured images, and automatically reconstruct the three-dimensional model of the object to be reconstructed according to the captured images under each perspective, without reconstructing the scene where the object to be reconstructed is located, so as to achieve the effect of separating the object to be reconstructed from the scene and modeling it separately. In addition, after determining the three-dimensional model of the object to be reconstructed, this embodiment will also perform coordinate conversion on the three-dimensional model, and then align the bottom center of the three-dimensional model to the origin of the world coordinate system, and align the y-axis of the three-dimensional model to the y-axis of the world coordinate system, so as to facilitate the user to preview the object to be reconstructed online and generate a rendering video around the object to be reconstructed.

[0056] The present invention aims to solve the problem that the target reconstructed object and the scene are coupled and difficult to separate. The 3D Gaussian representation 3D model of the target object is automatically reconstructed by the user inputting multi-view pictures taken around the object and the user clicking on the selected reconstructed object in one picture without reconstructing the scene. The center of the bottom surface of the modeled 3D model is aligned to the origin of the world coordinate system, and the y-axis is aligned to the y-axis of the world coordinate system, which can facilitate users to preview online, generate surround rendering videos, and subsequent video implantation and other applications.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions.

[0058] Based on this, the present application embodiment provides a three-dimensional modeling method, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the three-dimensional modeling method of the present application. In this embodiment, the method includes steps S10 to S30:

[0059] Step S10, acquiring photographic images containing the object to be reconstructed from multiple perspectives, and determining the camera shooting parameters of each photographic image, and performing point cloud reconstruction according to each photographic image to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture;

[0060] It should be noted that the images captured under multiple perspectives can be captured around the object to be reconstructed, or around any object, or any target in the scene where the object to be reconstructed is located. The target can be an object or a point in the scene, etc. The captured image shows the object to be reconstructed, and the morphology of the object to be reconstructed represented in the images captured under different perspectives may be different. The object to be reconstructed is an object that needs to be reconstructed in three dimensions. The object to be reconstructed can be any object in the scene corresponding to the captured image. The scene corresponding to the captured image and the scene where the object to be reconstructed is located are the same scene.

[0061] The camera shooting parameters include the camera pose, which can reflect the position and orientation of the camera when the camera is shooting around the target. The camera shooting parameters also include the camera internal parameters, which include the focal length, distortion coefficient and principal point coordinates of the camera. In this embodiment, the images containing the object to be reconstructed can be taken by the same camera, and the camera internal parameters of the same camera are generally fixed. Sparse point cloud data can reflect the three-dimensional framework of the object to be reconstructed and the scene in the three-dimensional space.

[0062] Exemplarily, images captured from multiple perspectives around the object to be reconstructed are obtained, the corresponding camera shooting parameters in each captured image are determined, and point cloud reconstruction is performed based on each captured image to obtain sparse point cloud data. In other embodiments, the motion structure recovery technology (SFM, Structure From Motion) can be used to determine the camera shooting parameters corresponding to each captured image and determine the sparse point cloud data. The camera intrinsic parameters of the images captured from different perspectives are the same, but the camera poses are different.

[0063] In a feasible embodiment, step S10 further includes steps S11 to S14:

[0064] Step S11, calling a preset feature extraction model to determine similar feature points between at least two captured images;

[0065] Step S12, extracting target feature points that match similar feature points from each captured image;

[0066] It should be noted that the preset feature extraction model can be a neural network model, and the preset feature extraction models can include: Netvlad (a vector descriptor based on a convolutional neural network), SuperPoint (a feature extractor based on a convolutional neural network) and SuperGlue (a feature matching method based on a graph neural network); similar feature points are image feature points with high similarity between each captured image.

[0067] Exemplarily, the global features of each captured image can be calculated by Netvlad, and the captured images to be matched can be determined in each captured image according to the global features, local feature points can be extracted from the captured images to be matched by SuperPoint, and local feature matching can be performed on at least two captured images to be matched by SuperGlue, and local feature points whose similarity between the two captured images is greater than a preset similarity threshold can be used as similar feature points. Among them, the global feature represents the vector or descriptor of the characteristics of the entire captured image, and the global feature can effectively distinguish the differences between images, so that the global feature can be used to screen out the captured images to be matched with smaller differences in each captured image, thereby facilitating the subsequent determination of similar feature points, and the local feature points are used to describe the feature description and coordinates of specific points (corner points, edge points) in the captured image. SuperPoint can generate a corresponding feature vector for each local feature point, so that when performing local feature matching on the two captured images to be matched, the similarity between the local features can be calculated by the respective feature vectors of the two local feature points, and then the local feature points whose similarity is greater than the preset similarity threshold can be used as similar feature points. There are multiple similar feature points between the two captured images to be matched.

[0068] Step S13, determining the camera shooting parameters corresponding to each captured image based on the coordinates of the target feature points corresponding to each captured image;

[0069] Step S14, mapping the target feature points corresponding to each captured image to a three-dimensional space to obtain sparse point cloud data.

[0070] It should be noted that each captured image has one or more corresponding target feature points. The camera pose corresponding to the captured image can be calculated based on the coordinates of the target feature points in the captured image, and the camera intrinsic parameters can also be determined. Based on the target feature points corresponding to each captured image, motion structure recovery can be performed to obtain sparse point cloud data and camera shooting parameters of each captured image.

[0071] Exemplarily, the target feature points in each captured image may be restored from motion in COLMAP (Collection of Mapping from Motion algorithms) to obtain camera intrinsic parameters, sparse point cloud data, and the camera pose of each captured image. For example, 1. Starting from a small number of captured images (for example, two captured images) and corresponding target feature points, the preliminary 3D structure and the camera pose corresponding to the captured images are restored by triangulating the captured images, wherein the 3D structure refers to the structure of the scene corresponding to the captured images and the object to be reconstructed in the 3D space; 2. Gradually add new captured images and corresponding target feature points to the existing 3D structure, and for each added captured image, the PnP (Perspective-n-Point) algorithm can be used to estimate the pose of the captured image, calculate the camera pose of the captured image, and determine the 3D points of the target feature points in the captured image on the 3D structure to map the target feature points to the 3D space, and perform bundle adjustment (BA) on the 3D structure to optimize the 3D structure; 3. Repeat step 2 above until all captured images are added to the 3D structure, obtain the camera pose of each captured image, and use the 3D structure as sparse point cloud data, wherein in COLMAP, the camera intrinsic parameters can usually be obtained through the EXIF ​​information (Exchangeable Image File) in the captured image. EXIF information can be obtained by using the Exchangeable Image File Format (EXIF). EXIF ​​information can record various shooting parameters of the captured image, such as shooting time, camera model, aperture value, shutter speed, focal length, etc.

[0072] This embodiment determines the camera pose and camera internal parameters of the captured image, and determines the sparse point cloud data, so as to facilitate the subsequent restoration of the object to be reconstructed in the captured image to the sparse point cloud in three-dimensional space based on the camera pose, thereby facilitating the subsequent three-dimensional reconstruction. In other embodiments, after the sparse point cloud data is determined, it will also be geometrically verified to facilitate the reconstruction quality of the sparse point cloud data. For example, the reconstructed sparse point cloud data can be presented using a three-dimensional visualization artifact to visually observe whether the sparse point cloud data completely covers the object to be reconstructed and the scene where the object to be reconstructed is located. This embodiment does not specifically limit the verification method.

[0073] For a better understanding of this embodiment, refer to Figure 2, the process of this embodiment is explained, wherein A1 refers to multiple captured images, A2 is a representation of sparse point cloud data, similar feature points can be determined through at least two of the multiple captured images, feature matching is performed based on the similar feature points, and corresponding target feature points are extracted from each captured image respectively, and corresponding camera shooting parameters are determined based on the corresponding target feature points of each captured image. At the same time, point clouds can also be reconstructed based on the target feature points of each captured image and in combination with the camera shooting parameters to obtain sparse point cloud data. The above is just one example, and this embodiment does not make any specific limitation on this.

[0074] Step S20, segmenting the object to be reconstructed in each captured image to obtain a reconstructed image of the object to be reconstructed corresponding to each captured image;

[0075] It should be noted that each captured image has its own corresponding reconstructed image, and the reconstructed image can be used to mark the position of the object to be reconstructed in the captured image. For example, in this embodiment, the object to be reconstructed in the captured image can be separated from the scene where the object to be reconstructed is located by a mask. The mask Mask can be a matrix of the same size as the image. Each element in the matrix can correspond to a pixel in the captured image, and the mask can be used to segment the image. The reconstructed image is characterized by the object to be reconstructed segmented from the captured image. Each reconstructed image includes a mask image and an RGB (Red, Green, Blue) image. The mask image records the mask information of the object to be reconstructed in the captured image, and the RGB image records the RGB information of the object to be reconstructed in the captured image. The mask information is used to mark the area where the object to be reconstructed is located. The pixel points in the area where the object to be reconstructed is located in the mask information can be marked as 1, and the pixel points in the area where the object to be reconstructed is not located can be marked as 0.

[0076] The object to be reconstructed may be selected by the user. For example, based on the object to be reconstructed selected by the user, the object to be reconstructed in each captured image may be segmented to obtain a reconstructed image of the object to be reconstructed corresponding to each captured image, so as to obtain a reconstructed image of the object to be reconstructed at different viewing angles.

[0077] In a feasible embodiment, the reconstructed image includes a target reconstructed image; each captured image includes a target captured image and other captured images, and step S20 further includes steps S21 to S23:

[0078] Step S21, receiving a click instruction from a user, determining a target mask area where the object to be reconstructed in the target captured image is located from the click instruction, and determining a target reconstructed image of the object to be reconstructed in the target captured image based on the target mask area;

[0079] It should be noted that the click instruction may be initiated by the user, and the click instruction is used to specify the object in the target captured image that needs to be 3D reconstructed, that is, to specify the object to be reconstructed in the target captured image, and the target mask area in the captured image may be selected, and the target reconstructed image of the object to be reconstructed corresponding to the target captured image may be determined based on the target mask area. The target mask area is the area in the captured image where the object to be reconstructed is located.

[0080] The target shot image may be any shot image. When the user selects the object to be reconstructed in the target shot graphic, the user may click on the target mask area where the object to be reconstructed is located, or may first click on an area where the object to be reconstructed is not located, and then reversely select to determine the target mask area of ​​the object to be reconstructed, thereby obtaining a target reconstructed image of the object to be reconstructed in the target shot image. For example, the pixels in the target mask area in the target shot image may be marked as 1, and the pixels in the non-target mask area may be marked as 0, thereby obtaining a target mask. In this embodiment, the target mask area in the target shot image may be determined by positive selection and / or reverse selection to determine the object to be reconstructed in the target shot image.

[0081] Step S22, segmenting each other captured image according to the target mask area to obtain a reconstructed mask area for each other captured image;

[0082] Step S23 : determining the reconstructed images of the object to be reconstructed in each of the other captured images based on the reconstruction mask regions of the other captured images.

[0083] It should be noted that the other captured images are the captured images other than the target captured image, and the reconstruction mask area is the area where the object to be reconstructed is located in the other captured images. In this embodiment, the trackanything model can be used to segment each captured image and obtain the reconstructed image corresponding to each captured image. Among them, the trackanything model is used to quickly and accurately track and segment specific objects in the video.

[0084] Exemplarily, each captured image can be combined into a picture set, and the picture set can be input into the track anything model in the form of a video. The track anything model can segment the object to be reconstructed in each other captured image in the picture set frame by frame based on the target mask area of ​​the target captured image, and obtain the reconstruction mask area corresponding to each other captured image, and determine the reconstructed image of the object to be reconstructed in each other captured image based on the reconstruction mask area corresponding to each other captured image.

[0085] This embodiment segments the object to be reconstructed in the captured image and determines reconstructed images of the object to be reconstructed at different viewing angles, thereby segmenting the object to be reconstructed from the scene where the object to be reconstructed is located, and segmenting the object to be reconstructed and the scene, thereby facilitating the subsequent generation of a three-dimensional model in which the object to be reconstructed and the scene are separated in the three-dimensional reconstruction process.

[0086] Step S30 , performing three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the shooting parameters of each camera and each reconstructed image, to obtain a three-dimensional model of the object to be reconstructed.

[0087] It should be noted that three-dimensional reconstruction of the object to be reconstructed refers to generating a three-dimensional model of the object to be reconstructed. By combining the sparse point cloud data and each reconstructed image, the object to be reconstructed under different perspectives can be integrated into the sparse point cloud data, and because the points in the sparse point cloud data have corresponding opacity, the points in the area where the object to be reconstructed is located in the sparse point cloud data can be determined based on the reconstructed image, and the points in the sparse point cloud data that are not in the area where the object to be reconstructed is located can be determined, so that the opacity of the points in the area where the object to be reconstructed is not located can be adjusted, so that the opacity of the points in the area where the object to be reconstructed is not located is 0 or close to 0, and then the opacity of the points in the area where the object to be reconstructed is not located in the sparse point cloud data is 0 or close to 0, thereby separating the object to be reconstructed from the scene, so that the three-dimensional model displays the object to be reconstructed, and does not display the background or other objects that are not related to the object to be reconstructed.

[0088] Exemplarily, based on each captured image, each reconstructed image and each camera shooting parameter, three-dimensional reconstruction is performed on the object to be reconstructed in the sparse point cloud, and at the same time, the opacity of points that do not belong to the area where the object to be reconstructed is located is adjusted to a preset opacity value, and the preset opacity value is 0, so as to obtain a three-dimensional model of the object to be reconstructed.

[0089] Since the sparse point cloud data can be represented as a three-dimensional framework reflected by each captured image, and the camera shooting posture in the camera shooting parameters of each captured image can reflect the position of the camera when shooting the object to be reconstructed at different viewing angles, and since the reconstructed image is obtained based on the segmentation of the object to be reconstructed in each captured image, each reconstructed image can individually reflect the object to be reconstructed at different viewing angles, that is, the reconstructed image represents the object to be reconstructed, but not the scene where the object to be reconstructed is located. Furthermore, through the camera shooting parameters, sparse point cloud data, and each reconstructed image, the object to be reconstructed at different viewing angles can be integrated into the sparse point cloud data in the three-dimensional space to obtain the three-dimensional model of the object to be reconstructed, so that the three-dimensional model represents the representation of the object to be reconstructed in the three-dimensional space, but does not represent the scene where the object to be reconstructed is located (for example, the background and\or irrelevant objects, etc.), thereby reducing the data redundancy of the three-dimensional model.

[0090] In a feasible embodiment, step S30 further includes steps S31 to S34:

[0091] Step S31, performing Gaussian initialization on the sparse point cloud data to obtain an initial three-dimensional Gaussian;

[0092] It should be noted that Gaussian initialization of sparse point cloud data means that for each point in the sparse point cloud data, the coordinates of the point are used as the center point of the Gaussian distribution. It can be considered that each point in the sparse point cloud data is surrounded by a Gaussian body, which can be a Gaussian ellipsoid. The initial three-dimensional Gaussian can be considered as a three-dimensional representation of the object to be reconstructed and the scene where the object to be reconstructed is located.

[0093] Step S32, projecting the initial three-dimensional Gaussian onto a two-dimensional plane according to the shooting parameters of each camera, to obtain renderings corresponding to the initial three-dimensional Gaussian under the shooting parameters of each camera;

[0094] It should be noted that different camera shooting parameters correspond to different renderings. For a camera shooting parameter, the camera shooting parameter corresponds to a shooting angle. The camera shooting parameter may correspond to multiple Gaussian bodies in the initial three-dimensional Gaussian. When the initial three-dimensional Gaussian is projected onto a two-dimensional plane based on the camera shooting parameter, multiple two-dimensional Gaussian images are generated. The multiple two-dimensional Gaussian images are fused to obtain a rendering of the camera shooting parameter.

[0095] This embodiment projects the initial three-dimensional Gaussian onto a two-dimensional plane according to the shooting parameters of each camera, so that rendering images under different shooting angles can be realized, which is convenient for subsequent comparison of the rendering images with the corresponding reconstructed images, and then it is convenient to integrate the objects to be reconstructed under different shooting angles into the initial three-dimensional Gaussian to obtain the three-dimensional model of the object to be reconstructed. Since the camera intrinsic parameters describe the optical properties inside the camera, etc., these properties determine how to map points in three-dimensional space to a two-dimensional plane, it is necessary to combine the camera intrinsic parameters and the camera posture to project the initial three-dimensional Gaussian onto a two-dimensional plane.

[0096] Step S33, successively calculating the loss value between each rendering image and the corresponding reconstructed image, and successively updating the three-dimensional Gaussian parameters of the initial three-dimensional Gaussian according to the loss value, so as to successively update the initial three-dimensional Gaussian and obtain an updated three-dimensional Gaussian;

[0097] Step S34, based on the updated three-dimensional Gaussian, determine the three-dimensional model of the object to be reconstructed, wherein the loss value is determined based on the pixel-by-pixel loss, structural similarity loss and opacity loss between the rendering and the corresponding reconstructed image, and the three-dimensional Gaussian parameters include opacity parameters.

[0098] It should be noted that the loss value between the rendering and the corresponding reconstructed image can be calculated by a preset loss function, which includes a pixel-wise loss function, a structural similarity loss function, and an opacity loss function. The pixel-wise loss function (Pixel-wise Loss) is used to calculate the error between the corresponding pixel values ​​between the rendering and the corresponding RGB image in the reconstructed image pixel by pixel; the structural similarity loss function (Structural Similarity Index, SSIM) is used to measure the structural similarity between the rendering and the corresponding RGB image in the reconstructed image, and the structural similarity loss can be calculated by calculating the brightness, contrast, and structure information between the images. The opacity loss function is used to calculate the error between the opacity of the corresponding pixel points between the rendering and the mask image in the corresponding reconstructed image pixel by pixel, and the opacity loss is calculated.

[0099] Since the preset loss function also includes an opacity loss function, the preset loss function can be used to perform opacity supervision in the three-dimensional reconstruction stage of the object to be reconstructed, so as to segment the object to be reconstructed and the scene where the object to be reconstructed is located. The loss value is calculated by the preset loss function, and the three-dimensional Gaussian parameters of the initial three-dimensional Gaussian are updated, so that the opacity of points that do not belong to the area where the object to be reconstructed is located can be set to 0, thereby facilitating the segmentation between the object to be reconstructed and the scene, and generating an independent three-dimensional model of the object to be reconstructed.

[0100] The three-dimensional Gaussian parameters include opacity parameters, and may also include RGB parameters and covariance matrix; wherein, the covariance matrix is ​​a 3x3 matrix used to describe the correlation between the three dimensions (x, y, z) in the initial three-dimensional Gaussian and the variance in each dimension. The variance is the element on the diagonal of the covariance matrix, indicating the degree of discreteness of the data distribution in each dimension. The elements on the non-diagonal line of the covariance matrix are covariances, indicating the correlation between different dimensions. The opacity parameter is used to describe the opacity of each Gaussian body in the initial three-dimensional Gaussian. When the opacity is 1, it means completely opaque; when the opacity is 0, it means completely transparent. The RGB parameter can be used to describe the values ​​of the three colors red (Red), green (Green), and blue (Blue) corresponding to each Gaussian body in the initial three-dimensional Gaussian.

[0101] The shooting angles of the rendering and the corresponding reconstructed image are consistent, and the loss value between the rendering and the corresponding reconstructed image can be expressed as:

[0102]

[0103] Where L is the loss value, is the pixel-by-pixel loss, is the structural similarity loss, is the opacity loss, (1-λ), λ, λ α They are the weight coefficients corresponding to the pixel-by-pixel loss, structural similarity loss, and opacity loss, respectively. The value of the structural similarity loss is between 0 and 1, and the larger the value, the more similar the two images are. Each loss value is the pixel-by-pixel loss and structural similarity loss between the same rendering and the corresponding RGB image in the reconstructed image, as well as the opacity loss between the same rendering and the corresponding mask image in the reconstructed image. Specifically, each rendering can be divided into an opacity rendering and an RGB rendering. The pixel-by-pixel loss and structural similarity loss between the RGB rendering and the corresponding RGB image in the reconstructed image can be calculated. The opacity loss is calculated through the opacity rendering and the mask image in the reconstructed image. The pixel-by-pixel loss, structural similarity loss, and opacity loss are weightedly summed to obtain the loss value of the rendering.

[0104] In this embodiment, the loss value between each rendering image and the corresponding reconstructed image can be calculated one by one, and the three-dimensional Gaussian parameters of the initial three-dimensional Gaussian can be updated one by one according to the loss value. The three-dimensional Gaussian parameters are updated by gradient feedback, and then the initial three-dimensional Gaussian can be updated. After the loss value is calculated between the reconstructed images corresponding to all the rendering images and the three-dimensional Gaussian parameters are updated based on the loss value, an updated three-dimensional Gaussian is obtained, and then the three-dimensional model of the object to be reconstructed can be determined based on the updated three-dimensional Gaussian.

[0105] This embodiment implements opacity supervision on the three-dimensional reconstruction process of the object to be reconstructed by calculating the loss value between the rendering and the reconstructed image, and finally reduces the opacity of the points in the scene where the object to be reconstructed is located to 0 or close to 0, so that the three-dimensional model displays the object to be reconstructed but not the scene where the object to be reconstructed is located, thereby achieving segmentation between the object to be reconstructed and the scene.

[0106] For a better understanding of this embodiment, refer to Figure 3 , briefly describe the process of this embodiment, where B1 is the reconstructed image, B2 is the rendering, and the loss value between the rendering and the reconstructed image is calculated, including structural similarity loss, pixel-by-pixel loss, and opacity loss; the rendering projected by the initial three-dimensional Gaussian should be as close to the reconstructed image as possible, so that the three-dimensional model of the object to be reconstructed can more accurately restore the actual object to be reconstructed, so by calculating the loss value between the rendering and the reconstructed image, the three-dimensional Gaussian parameters of the initial three-dimensional Gaussian are updated by the loss value, so that the object to be reconstructed represented by the updated three-dimensional Gaussian can be close to the real object to be reconstructed. In this example, the object to be reconstructed can be a cup.

[0107] Step S34 also includes step S341: eliminating target points whose opacity in the updated three-dimensional Gaussian is less than a preset opacity threshold, to obtain a three-dimensional model of the object to be reconstructed.

[0108] It should be noted that there may still be points with opacity close to 0 in the updated three-dimensional Gaussian, and these points may not belong to the points in the object to be reconstructed. Therefore, in order to ensure the accuracy of the three-dimensional model of the object to be reconstructed, the target points with opacity less than the preset opacity threshold in the updated three-dimensional Gaussian can be eliminated to obtain the three-dimensional model of the object to be reconstructed, thereby ensuring the accuracy of the three-dimensional reconstruction of the object to be reconstructed.

[0109] In a feasible embodiment, after step S30, steps A10 to A40 are also included:

[0110] Step A10, projecting the reconstructed points in the three-dimensional model to a world point cloud based on a world coordinate system to obtain a world point cloud including the reconstructed point cloud, and filtering out scene point clouds that do not match the reconstructed point cloud in the world point cloud;

[0111] Step A20, determining the target bottom surface in the scene point cloud, and generating a Gaussian dense point cloud bounding box of the object to be reconstructed in the world point cloud based on the normal vector of the target bottom surface and the target bottom surface, and determining the bottom center coordinates of the target bottom surface;

[0112] Step A30, determining a translation vector according to the relative position between the center coordinates of the bottom surface and the origin coordinates of the world coordinate system;

[0113] Step A40, performing coordinate transformation on the Gaussian dense point cloud bounding box and the points within the Gaussian dense point cloud bounding box according to a preset rotation matrix and a translation vector, to obtain a world three-dimensional model of the object to be reconstructed in a world coordinate system.

[0114] It should be noted that the reconstruction points may be points in a three-dimensional model, and each point in the three-dimensional model collectively represents a three-dimensional representation of the object to be reconstructed. The world point cloud is a point cloud based on a world coordinate system, and the origin of the world point cloud is the origin of the world coordinate system. The reconstruction points in the three-dimensional model may be projected to the world point cloud, so that the world point cloud includes the reconstruction point cloud, and the reconstruction point cloud is the common representation of each reconstruction point in the world point cloud, and the scene point cloud is the point cloud in the world point cloud except the reconstruction point cloud. The reconstructed point cloud may be labeled in the world point cloud. In other embodiments, key reconstruction points may be determined in the three-dimensional model to project the key reconstruction points into the world point cloud, and the key reconstruction points may be edge points in the three-dimensional reconstruction model, etc.

[0115] For example, the points corresponding to the reconstructed feature points in the world point cloud are marked as 1, while the points in the world point cloud that have not received the projection of the reconstructed feature points are marked as 0, so that the scene point cloud and the reconstructed point cloud can be distinguished in the world point cloud through the marked labels.

[0116] The target bottom surface refers to the bottom surface of the object to be reconstructed, and the Gaussian dense point cloud bounding box refers to the outline of the target bottom surface. The Gaussian dense point cloud bounding box can be determined based on the normal vector of the target bottom surface. For example, the minimum bounding box of the Gaussian dense point cloud can be generated based on the normal vector n of the target bottom surface. The minimum bounding box is the Gaussian dense point cloud bounding box of the target bottom surface. The preset rotation matrix can be determined based on the normal vector of the target bottom surface and the y-axis of the world coordinate system. The direction of the normal vector n is transformed to the direction of the y-axis of the world coordinate system to determine the preset rotation matrix. After the coordinate direction conversion, the center coordinate of the bottom surface may not be the origin coordinate of the world coordinate system, so the center coordinate of the bottom surface can be translated to the origin coordinate of the world coordinate system by determining the translation vector, and the coordinates of other points in the three-dimensional model are transformed to obtain the world three-dimensional model of the object to be reconstructed in the world coordinate system. The world three-dimensional model refers to the three-dimensional object to be reconstructed in the world coordinate system.

[0117] Among them, the step of generating the minimum bounding box of the Gaussian dense point cloud based on the normal vector n of the bottom surface of the target includes: traversing the points in the reconstructed point cloud, and calculating the minimum and maximum values ​​in the x and y directions in the intermediate point cloud on the plane perpendicular to the normal vector n (i.e., the parallel plane of the bottom surface of the target), including four values ​​(x_min, x_max, y_min, y_max), (x_min, x_max, y_min, y_max) and the minimum value (z_min) and maximum value (z_max) in the direction of the normal vector n, determining the boundary of the Gaussian dense point cloud bounding box, and the bottom surface of the Gaussian dense point cloud bounding box coincides with the bottom surface of the target. The Gaussian dense point cloud bounding box facilitates the obvious distinction between the boundary of the reconstructed point cloud and the scene point cloud in the world point cloud.

[0118] Exemplary, for example, if the coordinates of the center of the bottom surface are (x0, y0, z0), then the determined translation vector t = (-x0, -y0, -z0), wherein the coordinate transformation process is to first rotate and then translate, and the coordinate transformation of the Gaussian dense point cloud bounding box and the points within the Gaussian dense point cloud bounding box can be expressed as: X' = RX, X" = X' + t.

[0119] Among them, X can be used to represent the Gaussian dense point cloud bounding box and the coordinates in the Gaussian dense point cloud bounding box, X' is the rotated X, and X" is the translated X'. t is the translation vector, and R is the rotation matrix. By performing coordinate transformation on the Gaussian dense point cloud bounding box and the points in the Gaussian dense point cloud bounding box, the coordinate transformation of the three-dimensional model is achieved, and the representation of the three-dimensional model in the world coordinate system is obtained, that is, the world three-dimensional model of the object to be reconstructed in the world coordinate system is obtained. The Gaussian dense point cloud frame contains the three-dimensional model of the object to be reconstructed.

[0120] Since the bottom center of the 3D model may not be aligned with the coordinate origin of the world coordinate system, the y-axis of the 3D model will not be aligned with the y-axis of the world coordinate system, which will significantly affect the user's preview experience and affect the application of the 3D model in subsequent video rendering, etc. Specifically, when the user previews the 3D model through GaussianSplats3D (Gaussian renderer) based on JavaScript (a lightweight, interpreted or just-in-time compiled programming language with function priority), the 3D model will be in a random position and posture, and the user needs to spend a lot of time adjusting the viewing angle to obtain the front view of the 3D model. In addition, when generating a rotating rendering video, the camera rotates around the y-axis of the world coordinate system and renders each frame of the image. If the y-axis of the 3D model is not aligned with the y-axis of the world coordinate system, the rendering result will not be able to show the appearance of the 3D model well. At the same time, the 3D model that is not aligned with the bottom center and the y-axis of the world coordinate system cannot be directly applied to applications such as VR (Virtual Reality), AR (Augmented Reality) or video implantation. To this end, this embodiment performs coordinate transformation on the three-dimensional model by adjusting the three-dimensional model. For example, the target bottom surface of the three-dimensional model is first determined, and then the coordinate transformation of the three-dimensional model is performed to obtain a world three-dimensional model of the object to be reconstructed in the world coordinate system, thereby facilitating improving the user's preview experience.

[0121] In a feasible embodiment, step A30 further includes steps A31 and A32:

[0122] Step A31, based on a preset random sampling algorithm, plane estimation is performed in the scene point cloud to obtain multiple candidate bottom surfaces;

[0123] Step A32, determining a target bottom surface that meets a preset bottom surface condition from among the candidate bottom surfaces;

[0124] The preset bottom surface conditions include: the average distance between each preset reconstruction point in the reconstructed point cloud and the candidate bottom surface is less than the preset point cloud span threshold, the difference between the number of points in the point clouds on both sides of the candidate bottom surface is less than the preset first point cloud threshold, and the number of points in the candidate bottom surface is greater than the preset second point cloud threshold.

[0125] It should be noted that the preset random sampling algorithm refers to RANSAC (Random Sample Consensus), which can perform plane estimation in the scene point cloud to obtain multiple candidate bottom surfaces. The number of preset reconstruction points in the reconstructed point cloud can be a preset number, and the preset number of preset reconstruction points are the first preset number of points in the reconstructed point cloud that are closest to the candidate bottom surface in vertical distance. The number of points in the candidate bottom surface refers to the number of all points existing in the candidate bottom surface. Different candidate bottom surfaces have different preset reconstruction points in the reconstructed point cloud.

[0126] The preset point cloud span threshold is the product of the maximum span of coordinates in the reconstructed point cloud and the preset first coefficient, and the preset first coefficient can be 0.2; the step of determining the maximum span of coordinates can include: calculating the first absolute value of the difference between the minimum x coordinate and the maximum x coordinate of the reconstructed point cloud in the x-axis direction, calculating the second absolute value of the difference between the minimum y coordinate and the maximum y coordinate of the reconstructed point cloud in the y-axis direction, and calculating the third absolute value of the difference between the minimum z coordinate and the maximum z coordinate of the reconstructed point cloud in the y-axis direction; selecting the largest value among the first absolute value of the difference, the second absolute value of the difference, and the third absolute value of the difference as the maximum span of coordinates. The preset first point cloud threshold is the product of the number of points in the reconstructed point cloud and the preset second coefficient, and the preset second coefficient can be 0.03; the preset second point cloud threshold is the product of the number of points in the reconstructed point cloud and the preset third coefficient, and the preset third coefficient can be 0.001.

[0127] Among them, when the average distance between each preset reconstruction point in the reconstructed point cloud and the target bottom surface is less than the preset point cloud span threshold, it means that the target bottom surface is closest to the foreground and the target bottom surface is most likely to coincide with the foreground; when the difference between the number of points in the point cloud on both sides of the target bottom surface is less than the preset first point cloud threshold, it means that the foreground is on one side of the bottom surface; when the number of points in the target bottom surface is greater than the preset second point cloud threshold, it means that the most foreground points are retained on the target bottom surface.

[0128] Exemplarily, if the candidate bottom surface satisfies the following conditions at the same time: the average distance between the candidate bottom surface and each preset reconstruction point in the reconstructed point cloud is less than the preset point cloud span threshold, the difference between the number of points in the point clouds on both sides of the candidate bottom surface is less than the preset first point cloud threshold, and the number of points in the candidate bottom surface is greater than the preset second point cloud threshold, the candidate bottom surface is determined to be the target bottom surface. If there are multiple candidate bottom surfaces that meet the preset bottom surface conditions, the candidate bottom surface with the smallest difference between the number of points in the point clouds on both sides of the candidate bottom surface is selected as the target bottom surface from the multiple candidate bottom surfaces that meet the preset bottom surface conditions. This embodiment can improve the accuracy of determining the target candidate bottom surface.

[0129] For a better understanding of this embodiment, refer to Figure 4, the overall process of this embodiment is illustrated by an example. In this example, the object to be reconstructed may be a cup, A1 is a plurality of photographed images, and motion structure recovery SFM is performed on the plurality of photographed images to obtain sparse point cloud data and camera shooting parameters of each photographed image. The plurality of photographed images are segmented to determine the mask of the object to be reconstructed in each photographed image, so as to determine the reconstructed image corresponding to each photographed image. Based on the reconstructed image, the shooting parameters of each camera and the sparse point cloud data, the object to be reconstructed can be reconstructed in three dimensions to obtain a three-dimensional model of the object to be reconstructed. Then, the coordinate transformation of the three-dimensional model can be performed to determine the representation of the three-dimensional model in the world coordinate system to obtain a world three-dimensional model. For example, the reconstructed points in the three-dimensional model are projected to the world point cloud based on the world coordinate system to obtain a world point cloud containing the reconstructed point cloud, and the scene point cloud that does not match the reconstructed point cloud is screened out in the world point cloud; the target bottom surface is determined in the scene point cloud, and based on the normal vector of the target bottom surface and the target bottom surface, a Gaussian dense point cloud bounding box of the object to be reconstructed is generated in the world point cloud, and the bottom center coordinates of the target bottom surface are determined; the translation vector is determined according to the relative position between the bottom center coordinates and the origin coordinates of the world coordinate system; the Gaussian dense point cloud bounding box and the points in the Gaussian dense point cloud bounding box are transformed according to the preset rotation matrix and the translation vector to obtain a world three-dimensional model of the object to be reconstructed in the world coordinate system.

[0130] This application also provides a three-dimensional modeling device, please refer to Figure 5 , the three-dimensional modeling device includes:

[0131] An acquisition module 10 is used to acquire captured images containing the object to be reconstructed from multiple perspectives, and respectively determine the camera shooting parameters of each captured image, and perform point cloud reconstruction according to each captured image to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture;

[0132] The segmentation module 20 is used to segment the object to be reconstructed in each captured image to obtain a reconstructed image of the object to be reconstructed corresponding to each captured image;

[0133] The reconstruction module 30 is used to perform three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the shooting parameters of each camera and each reconstructed image, so as to obtain a three-dimensional model of the object to be reconstructed.

[0134] The three-dimensional modeling device provided in the present application adopts the three-dimensional modeling method in the above-mentioned embodiment, aiming to solve the technical problem of high data redundancy of the reconstructed three-dimensional model. Compared with the prior art, the beneficial effects of the three-dimensional modeling method provided in the embodiment of the present application are the same as the beneficial effects of the three-dimensional modeling method provided in the above-mentioned embodiment, and other technical features in the three-dimensional modeling device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here. The embodiment of the present application provides an electronic device, which may be a playback device, and the electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that at least one processor can execute the three-dimensional modeling method in the above-mentioned embodiment.

[0135] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] like Figure 6 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. In RAM 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device may allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0137] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0138] The electronic device provided by the present application adopts the three-dimensional modeling method in the above-mentioned embodiment 1 to solve the technical problem of high data redundancy of the reconstructed three-dimensional model. Compared with the prior art, the beneficial effect of the product flow data distribution provided by the embodiment of the present application is the same as the beneficial effect of the three-dimensional modeling method provided by the above-mentioned embodiment, and the other technical features in the three-dimensional modeling device are the same as the features disclosed by the above-mentioned embodiment method, which will not be repeated here. It should be understood that the various parts of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples. The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. The present embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the three-dimensional modeling method in the above-mentioned embodiment 1. The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, equipment or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable EPROM (Electrical Programmable Read Only Memory) or flash memory, an optical fiber, a portable compact disk CD-ROM (compact disc read-only memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution device, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above. The above-mentioned computer-readable storage medium may be contained in an electronic device; it may also exist alone without being assembled into an electronic device.The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: obtains captured images containing objects to be reconstructed from multiple perspectives, and determines the camera shooting parameters of each of the captured images respectively, and reconstructs point clouds according to each of the captured images to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture; segments the objects to be reconstructed in each of the captured images respectively to obtain reconstructed images of the objects to be reconstructed corresponding to each of the captured images; and performs three-dimensional reconstruction of the objects to be reconstructed according to the sparse point cloud data, the camera shooting parameters and the reconstructed images to obtain a three-dimensional model of the objects to be reconstructed.

[0139] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a LAN (local area network) or WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The name of the module does not constitute a limitation on the unit itself under certain circumstances. The computer-readable storage medium provided in the present application stores computer-readable program instructions for executing the above-mentioned three-dimensional modeling method, aiming to solve the technical problem of high data redundancy of the reconstructed three-dimensional model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiments of the present application are the same as the beneficial effects of the three-dimensional modeling method provided in the above-mentioned embodiments, which will not be repeated here.

[0142] The present application also provides a computer program product, including a computer program, which implements the steps of the three-dimensional modeling method as described above when executed by a processor. The computer program product provided by the present application is intended to solve the technical problem of high data redundancy of the reconstructed three-dimensional model. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the three-dimensional modeling method provided by the above-mentioned embodiment, and are not repeated here. The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structures or equivalent process transformations made using the contents of the specification and drawings of this application, or directly or indirectly used in other related technical fields, are similarly included in the patent processing scope of the present application.

Claims

1. A three-dimensional modeling method, characterized in that: The method includes: Acquire captured images containing the object to be reconstructed from multiple perspectives, determine the camera shooting parameters of each of the captured images, and perform point cloud reconstruction according to each of the captured images to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture; Segmenting the objects to be reconstructed in the captured images respectively to obtain reconstructed images of the objects to be reconstructed corresponding to the captured images; The object to be reconstructed is three-dimensionally reconstructed according to the sparse point cloud data, the shooting parameters of each camera and the reconstructed images to obtain a three-dimensional model of the object to be reconstructed.

2. The method according to claim 1, characterized in that The steps of respectively determining the camera shooting parameters of each of the captured images, and reconstructing the point cloud according to each of the captured images to obtain sparse point cloud data include: Calling a preset feature extraction model to determine similar feature points between at least two of the captured images; Extracting target feature points that match the similar feature points from each of the captured images; Determining the camera shooting parameters corresponding to each of the captured images based on the coordinates of the target feature points corresponding to each of the captured images; The target feature points corresponding to each of the captured images are mapped to three-dimensional space to obtain sparse point cloud data.

3. The method according to claim 1, characterized in that The reconstructed image includes a target reconstructed image; each of the captured images includes a target captured image and other captured images, and the steps of segmenting the objects to be reconstructed in each of the captured images to obtain the reconstructed images of the objects to be reconstructed corresponding to each of the captured images include: receiving a click instruction from a user, determining a target mask area of ​​the object to be reconstructed in the target captured image from the click instruction, and determining a target reconstructed image of the object to be reconstructed in the target captured image based on the target mask area; According to the target mask area, each of the other captured images is segmented to obtain a respective reconstructed mask area of ​​each of the other captured images; Based on the reconstruction mask area of ​​each of the other captured images, a reconstructed image of the object to be reconstructed corresponding to each of the other captured images is determined.

4. The method according to claim 1, characterized in that The step of performing three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, each of the camera shooting parameters and each of the reconstructed images to obtain a three-dimensional model of the object to be reconstructed comprises: Performing Gaussian initialization on the sparse point cloud data to obtain an initial three-dimensional Gaussian; According to each of the camera shooting parameters, the initial three-dimensional Gaussian is projected onto a two-dimensional plane to obtain renderings corresponding to the initial three-dimensional Gaussian under each of the camera shooting parameters; Successively calculating the loss value between each of the rendering images and the corresponding reconstructed image, and successively updating the three-dimensional Gaussian parameters of the initial three-dimensional Gaussian according to the loss value, so as to successively update the initial three-dimensional Gaussian to obtain an updated three-dimensional Gaussian; Based on the updated three-dimensional Gaussian, a three-dimensional model of the object to be reconstructed is determined, wherein the loss value is determined based on pixel-by-pixel loss, structural similarity loss and opacity loss between the rendering and the corresponding reconstructed image, and the three-dimensional Gaussian parameters include opacity parameters.

5. The method according to claim 4, characterized in that The step of determining the three-dimensional model of the object to be reconstructed based on the updated three-dimensional Gaussian includes: Target points whose opacity is less than a preset opacity threshold in the updated three-dimensional Gaussian are eliminated to obtain the three-dimensional model of the object to be reconstructed.

6. The method according to claim 1, characterized in that After the step of performing three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the camera shooting parameters and the reconstructed images to obtain a three-dimensional model of the object to be reconstructed, the method further includes: Projecting the reconstructed points in the three-dimensional model to a world point cloud based on a world coordinate system to obtain a world point cloud including the reconstructed point cloud, and filtering out scene point clouds that do not match the reconstructed point cloud in the world point cloud; Determine a target bottom surface in the scene point cloud, and generate a Gaussian dense point cloud bounding box of the object to be reconstructed in the world point cloud based on a normal vector of the target bottom surface and the target bottom surface, and determine the bottom center coordinates of the target bottom surface; Determine a translation vector according to the relative position between the center coordinates of the bottom surface and the origin coordinates of the world coordinate system; According to the preset rotation matrix and translation vector, the Gaussian dense point cloud bounding box and the points within the Gaussian dense point cloud bounding box are transformed to obtain a world three-dimensional model of the object to be reconstructed in the world coordinate system.

7. The method according to claim 6, characterized in that The step of determining the bottom surface of the target in the scene point cloud comprises: Based on a preset random sampling algorithm, plane estimation is performed in the scene point cloud to obtain multiple candidate bottom surfaces; Determining a target bottom surface that meets a preset bottom surface condition among the candidate bottom surfaces; The preset bottom surface conditions include: the average distance between each preset reconstruction point in the reconstructed point cloud and the candidate bottom surface is less than a preset point cloud span threshold, the difference between the number of points in the point clouds on both sides of the candidate bottom surface is less than a preset first point cloud threshold, and the number of points in the candidate bottom surface is greater than a preset second point cloud threshold.

8. A three-dimensional modeling device, characterized in that: The device comprises: An acquisition module, used to acquire photographed images containing the object to be reconstructed from multiple perspectives, and respectively determine the camera shooting parameters of each of the photographed images, and perform point cloud reconstruction according to each of the photographed images to obtain sparse point cloud data, wherein the camera shooting parameters include camera posture; A segmentation module, used to segment the objects to be reconstructed in each of the captured images, respectively, to obtain reconstructed images of the objects to be reconstructed corresponding to each of the captured images; A reconstruction module is used to perform three-dimensional reconstruction on the object to be reconstructed according to the sparse point cloud data, the camera shooting parameters and the reconstructed images to obtain a three-dimensional model of the object to be reconstructed.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the three-dimensional modeling method described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium is the computer-readable storage medium, on which is stored a program for implementing the three-dimensional modeling method, and the program for implementing the three-dimensional modeling method is executed by a processor to implement the steps of the three-dimensional modeling method as described in any one of claims 1 to 7.

Images