Three-dimensional reconstruction method and device of single target, electronic equipment and storage medium

Abstract

The invention provides a three-dimensional reconstruction method and device of a single target, electronic equipment and a storage medium, and relates to the technical field of image three-dimensional reconstruction. The method comprises the following steps: inputting a pre-shot multi-view image and a preset cue word into a target visual segmentation model, and obtaining a pixel-level mask of a to-be-reconstructed single target output by the target visual segmentation model based on the multi-view image and the preset cue word; and then, mapping the pixel-level mask into a three-dimensional space point cloud, and carrying out clustering processing on the three-dimensional space point cloud based on a preset algorithm, thereby determining a geometric center point and a three-dimensional bounding box of the to-be-reconstructed single target. And further, inputting the geometric center point and the three-dimensional bounding box into the target three-dimensional reconstruction model to obtain a three-dimensional model of the to-be-reconstructed single target output by the target three-dimensional reconstruction model. According to the target three-dimensional reconstruction model, the accurate and credible three-dimensional model of the to-be-reconstructed single target can be output.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method and device for a single object, an electronic device and a storage medium. BACKGROUND

[0002] With the development of computer vision and three-dimensional modeling technology, three-dimensional reconstruction technology for a single object has been widely applied in many fields. The core of three-dimensional reconstruction technology is to restore the three-dimensional geometric structure and surface features of a single object from the collected two-dimensional images, point clouds and other data, and generate a three-dimensional model that can be used for subsequent analysis or interaction.

[0003] The existing three-dimensional reconstruction method for a single object uses a camera or other device to take 360° all-around pictures around the single object, to obtain multiple two-dimensional images of the single object under different shooting angles. At the same time, it is ensured that the images taken contain the complete appearance features of the single object, such as contour, texture, structural details, etc. Then, feature matching is performed on the multiple two-dimensional images, and the pose of the camera under each shooting angle, i.e. the position and orientation of the camera in space, is solved. Further, the center coordinates of the single object are calculated according to the pose of the camera. Taking the center coordinates of the single object as a reference, the three-dimensional coordinates of the single object are calculated in combination with the two-dimensional images and the pose of the camera under each shooting angle, and thus the three-dimensional reconstruction of the single object is completed.

[0004] However, if the single object is located at a corner of a wall or other location with limited viewing angle, the camera cannot take all-around pictures of the single object. At this time, the center coordinates of the single object cannot be accurately calculated according to the pose of the camera, and the three-dimensional coordinates of the single object obtained by taking the center coordinates as a reference are also inaccurate, which further leads to an inaccurate three-dimensional reconstruction result of the single object. SUMMARY

[0005] The present application provides a three-dimensional reconstruction method, device, electronic device and storage medium for a single object, to solve the technical problem that the three-dimensional model of a single object generated by the prior art is not accurate enough.

[0006] In a first aspect, the present application provides a three-dimensional reconstruction method for a single object, comprising:

[0007] inputting a plurality of perspective images and a preset prompt word into a target visual segmentation model, and obtaining a pixel-level mask of a single object to be reconstructed output by the target visual segmentation model based on the plurality of perspective images and the preset prompt word; wherein the plurality of perspective images represent a set of multiple images with different perspectives, each image included in the plurality of perspective images contains the single object to be reconstructed; and the preset prompt word is used to define the category to which the single object to be reconstructed belongs.

[0008] mapping the pixel-level mask into a three-dimensional spatial point cloud; performing clustering processing on the three-dimensional spatial point cloud based on a preset algorithm to determine a geometric center point and a three-dimensional bounding box of the single-body target to be reconstructed;

[0009] inputting the geometric center point and the three-dimensional bounding box into a target three-dimensional reconstruction model to obtain a three-dimensional model of the single-body target to be reconstructed output by the target three-dimensional reconstruction model; wherein the three-dimensional model represents a digital representation of a three-dimensional spatial structure of the single-body target to be reconstructed.

[0010] In a possible design, mapping the pixel-level mask into a three-dimensional spatial point cloud comprises:

[0011] adopting a preset pose estimation algorithm to calculate a translation vector and a rotation vector of a target camera in an image shooting process; determining external parameters of the target camera based on the translation vector and the rotation vector; wherein the target camera is used to shoot images containing the single-body target to be reconstructed from different perspectives; the translation vector indicates a spatial position of the target camera, the rotation vector indicates an orientation of the target camera, and the external parameters indicate a pose of the target camera;

[0012] obtaining internal parameters of the target camera; performing inverse projection processing on the pixel-level mask based on the external parameters and the internal parameters to map the pixel-level mask into a three-dimensional spatial point cloud; wherein the internal parameters indicate a projection relationship from a three-dimensional camera coordinate system to a two-dimensional image pixel coordinate system.

[0013] In a possible design, performing inverse projection processing on the pixel-level mask based on the external parameters and the internal parameters to map the pixel-level mask into a three-dimensional spatial point cloud comprises:

[0014] determining two-dimensional coordinates of pixels included in the pixel-level mask;

[0015] converting the two-dimensional coordinates into three-dimensional coordinate points based on the external parameters and the internal parameters; and generating a three-dimensional spatial point cloud based on the three-dimensional coordinate points.

[0016] In a possible design, the preset algorithm is an unsupervised spatial clustering algorithm;

[0017] performing clustering processing on the three-dimensional spatial point cloud based on a preset algorithm to determine a geometric center point and a three-dimensional bounding box of the single-body target to be reconstructed comprises:

[0018] based on the unsupervised spatial clustering algorithm, dividing the three-dimensional spatial point cloud into a plurality of point cloud subsets in combination with spatial distribution characteristics of the three-dimensional spatial point cloud;

[0019] From the plurality of point cloud subsets, a target point cloud subset containing the single-body target to be reconstructed is determined; based on three-dimensional coordinates of the target point cloud subset, a geometric center point and a three-dimensional bounding box of the single-body target to be reconstructed are calculated.

[0020] In a possible design, the geometric center point and the three-dimensional bounding box are input into a target three-dimensional reconstruction model, and a three-dimensional model of the single-body target to be reconstructed output by the target three-dimensional reconstruction model is obtained, including:

[0021] The geometric center point and the three-dimensional bounding box are input into a target three-dimensional reconstruction model.

[0022] Based on a preset sampling strategy and a preset spatial constraint inside the target three-dimensional reconstruction model, three-dimensional modeling is performed on the single-body target to be reconstructed in combination with the geometric center point and the three-dimensional bounding box.

[0023] A three-dimensional model of the single-body target to be reconstructed output by the target three-dimensional reconstruction model is obtained.

[0024] In a possible design, based on a preset sampling strategy and a preset spatial constraint inside the target three-dimensional reconstruction model, three-dimensional modeling is performed on the single-body target to be reconstructed in combination with the geometric center point and the three-dimensional bounding box, including:

[0025] Based on the geometric center point and the three-dimensional bounding box, a reconstruction region of the target three-dimensional reconstruction model is determined.

[0026] Based on the target three-dimensional reconstruction model, sampling points are generated in the reconstruction region; and based on the sampling points, three-dimensional modeling is performed on the single-body target to be reconstructed in the reconstruction region.

[0027] In a possible design, the target visual segmentation model obtains, based on the multi-view image and the preset prompt word, a pixel-level mask of the single-body target to be reconstructed, including:

[0028] Based on the target visual segmentation model, visual features of the multi-view image are determined by performing feature extraction on the multi-view image; and the preset prompt word is converted into a semantic vector.

[0029] The visual features and the semantic vector are matched, and a probability that each pixel in the multi-view image belongs to the single-body target to be reconstructed is predicted.

[0030] Pixels with a probability greater than or equal to a preset threshold are determined as the single-body target to be reconstructed, and pixels with a probability less than the preset threshold are determined as background.

[0031] The binary label is performed on the monomer target to be reconstructed and the background, and a pixel-level mask of the monomer target to be reconstructed is generated.

[0032] In a second aspect, the present application provides a three-dimensional reconstruction device of a monomer target, comprising:

[0033] An input module is configured to input a pre-shot multi-view image and a preset prompt word into a target visual segmentation model.

[0034] An acquisition module is configured to acquire a pixel-level mask of a monomer target to be reconstructed output by the target visual segmentation model based on the multi-view image and the preset prompt word.

[0035] A mapping module is configured to map the pixel-level mask into a three-dimensional space point cloud.

[0036] A processing module is configured to perform clustering processing on the three-dimensional space point cloud based on a preset algorithm to determine a geometric center point and a three-dimensional bounding box of the monomer target to be reconstructed.

[0037] The input module is further configured to input the geometric center point and the three-dimensional bounding box into a target three-dimensional reconstruction model to obtain a three-dimensional model of the monomer target to be reconstructed output by the target three-dimensional reconstruction model.

[0038] In a possible design, the mapping module further comprises a calculation module and a determination module.

[0039] The calculation module is configured to calculate a translation vector and a rotation vector of a target camera in an image shooting process by using a preset pose estimation algorithm.

[0040] The determination module is configured to determine an external parameter of the target camera based on the translation vector and the rotation vector.

[0041] The acquisition module is further configured to acquire an internal parameter of the target camera.

[0042] The processing module is further configured to perform inverse projection processing on the pixel-level mask based on the external parameters and the internal parameters, so as to map the pixel-level mask to a point cloud in a three-dimensional space, wherein the internal parameters indicate a projection relationship from a three-dimensional camera coordinate system to a two-dimensional image pixel coordinate system.

[0043] In a possible design, the determining module is further configured to determine that the pixel-level mask includes two-dimensional coordinates of pixels.

[0044] The processing module further includes a conversion module and a generation module.

[0045] The conversion module is configured to convert the two-dimensional coordinates to three-dimensional coordinate points based on the external parameters and the internal parameters.

[0046] The generation module is configured to generate a point cloud in a three-dimensional space based on the three-dimensional coordinate points.

[0047] In a possible design, the preset algorithm is an unsupervised spatial clustering algorithm.

[0048] The processing module further includes a division module configured to divide, based on the unsupervised spatial clustering algorithm, the point cloud in the three-dimensional space into a plurality of point cloud subsets in combination with spatial distribution characteristics of the point cloud in the three-dimensional space.

[0049] The determining module is further configured to determine, from the plurality of point cloud subsets, a target point cloud subset containing the single-body target to be reconstructed.

[0050] The calculation module is further configured to calculate a geometric center point and a three-dimensional bounding box of the single-body target to be reconstructed based on three-dimensional coordinates of the target point cloud subset.

[0051] In a possible design, the input module is further configured to input the geometric center point and the three-dimensional bounding box to a target three-dimensional reconstruction model.

[0052] The processing module is further configured to perform three-dimensional modeling on the single-body target to be reconstructed based on a preset sampling strategy and a preset spatial constraint inside the target three-dimensional reconstruction model in combination with the geometric center point and the three-dimensional bounding box.

[0053] The obtaining module is further configured to obtain a three-dimensional model of the single-body target to be reconstructed output by the target three-dimensional reconstruction model.

[0054] In a possible design, the determining module is further configured to determine a reconstruction region of the target three-dimensional reconstruction model based on the geometric center point and the three-dimensional bounding box.

[0055] The processing module further includes a sampling module configured to sample in the reconstruction region based on the target three-dimensional reconstruction model to generate corresponding sampling points.

[0056] The processing module is further configured to perform three-dimensional modeling on the single target to be reconstructed in the reconstruction region based on the sampling points.

[0057] In a possible design, the obtaining module further includes an extracting module, a matching module, and a marking module.

[0058] The extracting module is configured to perform feature extraction on the multi-view images based on the target visual segmentation model to determine visual features of the multi-view images.

[0059] The conversion module is further configured to convert the preset prompt word into a semantic vector.

[0060] The matching module is configured to match the visual features with the semantic vector to predict a probability that each pixel in the multi-view images belongs to the single target to be reconstructed.

[0061] The determining module is further configured to determine, as the single target to be reconstructed, a pixel whose probability is greater than or equal to a preset threshold, and determine, as a background, a pixel whose probability is less than the preset threshold.

[0062] The marking module is configured to perform binary marking on the single target to be reconstructed and the background to generate a pixel-level mask of the single target to be reconstructed.

[0063] In a third aspect, an electronic device is provided, including at least one processor and a memory; the memory stores computer-executed instructions; the at least one processor executes the computer-executed instructions stored in the memory, so that the at least one processor performs the method in the first aspect and various possible designs.

[0064] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executed instructions. When a processor executes the computer-executed instructions, the method in the first aspect and various possible designs is implemented.

[0065] In a fifth aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method in the first aspect and various possible designs is implemented.

[0066] The method, apparatus, electronic device, and storage medium for 3D reconstruction of a single target provided in this application input pre-captured multi-view images and preset prompts into a target visual segmentation model, and obtain the pixel-level mask of the single target to be reconstructed, output by the target visual segmentation model based on the multi-view images and preset prompts. The multi-view images represent a set of images with different perspectives, each containing the single target to be reconstructed. The preset prompts define the category of the single target. The pixel-level mask is then mapped to a 3D point cloud, realizing the transformation of 2D image information into potential positional constraints in 3D space. Even with limited perspectives, this provides basic clues for locating the single target. Furthermore, a preset algorithm is used to cluster the 3D point cloud to determine the geometric center point and 3D bounding box of the single target. The clustering process does not rely on the integrity of the perspective; by mining the spatial correlation of the 3D point cloud, point clouds belonging to the same single target are clustered together, accurately defining the spatial range of the single target even with incomplete perspectives. Finally, the geometric center point and 3D bounding box are input into the target 3D reconstruction model to obtain the 3D model of the single target to be reconstructed, output by the target 3D reconstruction model. This demonstrates that regardless of whether the shooting angle corresponding to the multi-view images is limited, the geometric center point and 3D bounding box of the single target to be reconstructed can be accurately inferred. Based on the accuracy of the geometric center point and 3D bounding box, the 3D model of the single target to be reconstructed output by the target 3D reconstruction model is also accurate and reliable. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0068] Figure 1 A flowchart illustrating the three-dimensional reconstruction method for a single target provided in this application embodiment. Figure One ;

[0069] Figure 2 A schematic diagram of the shooting scene for the three-dimensional reconstruction method of a single target provided in the embodiments of this application. Figure One ;

[0070] Figure 3 A schematic diagram of the shooting scene for the three-dimensional reconstruction method of a single target provided in the embodiments of this application. Figure Two ;

[0071] Figure 4 A flowchart illustrating the three-dimensional reconstruction method for a single target provided in this application embodiment. Figure Two ;

[0072] Figure 5A structural schematic diagram of a three-dimensional reconstruction device of a single target provided by an embodiment of the present application is shown in the figure.

[0073] Figure 6 A hardware structural diagram of an electronic device provided by an embodiment of the present application is shown in the figure.

[0074] The specific embodiments of the present application have been shown in the above figures, and will be described in more detail hereinafter. These figures and the written description are not intended to limit the scope of the present application in any way, but to illustrate the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0075] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The embodiments described in the following exemplary embodiments are not meant to represent the only implementations in which the present application can be practiced. Rather, the embodiments are provided as examples of apparatus and methods of practicing the present application as detailed in the following claims.

[0076] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, and the above-described figures, if any, are used to distinguish between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, where appropriate, to refer to an embodiment of the application described herein in order to, for example, enable an embodiment of the application described herein to be practiced in other than the order described or indicated herein.

[0077] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean example, illustration, or instance, and do not necessarily pertain to a preference or a better, more advantageous, or more functional implementation. In the description of embodiments of the present application, any example, or embodiment and design solution described as "exemplary" or "for example" should not be construed as preferable or more advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present concepts in a particular manner.

[0078] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the relevant data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for the user to choose authorization or rejection.

[0079] Three-dimensional reconstruction refers to establishing a mathematical model suitable for computer representation and processing for a three-dimensional object (single object target), which is the basis for processing, operating and property analysis of the three-dimensional object in a computer environment, and is also a key technology for establishing a virtual reality representing the real world in a computer.

[0080] The existing three-dimensional reconstruction method of a single object target adopts a camera or other equipment to perform 360° omnidirectional shooting around the single object target to obtain multiple two-dimensional images of the single object target at different shooting angles. Meanwhile, it ensures that the images contain complete appearance features of the single object target, such as contour, texture, structural details, etc. Then, feature matching is performed on the multiple two-dimensional images to solve the pose of the camera at each shooting angle, i.e., the position and orientation of the camera in space. Further, the center coordinates of the single object target are calculated according to the pose of the camera. With the center coordinates of the single object target as a reference, the three-dimensional coordinates of the single object target are calculated in combination with the two-dimensional images and the pose of the camera at each shooting angle, and thus the three-dimensional reconstruction of the single object target is completed.

[0081] However, in the case that the single object target is located at a corner of a wall or other angle-limited position, the shooting angle of the camera is limited (e.g., only front and side), and the single object target cannot be shot omnidirectionally. At this time, the overlapping area between the multiple two-dimensional images shot by the camera is extremely small, and lacks non-coplanar feature points, which leads to errors in feature matching of the multiple two-dimensional images and makes it impossible to accurately calculate the pose of the camera at each shooting angle.

[0082] In addition, angle limitation not only affects the accuracy of the pose of the camera, but also leads to the loss of three-dimensional geometric information of the single object target. This is because the part of the single object target that is blocked is completely invisible in the two-dimensional image sequence, and lacks any pixel features for learning.

[0083] Therefore, the calculated center coordinates of the single object target deviate from the actual center position of the single object target. On the basis of inaccurate center coordinates of the single object target, the three-dimensional coordinates of the single object target obtained with the center coordinates as a reference are also inaccurate, which finally leads to an inaccurate three-dimensional shape of the single object target constructed according to the three-dimensional coordinates.

[0084] In view of the above technical problems, in order to accurately calculate the pose of the camera at each shooting angle, it is necessary to rely on the all-around shooting of the camera. Based on this, the inventors have realized that the two-dimensional image usually contains three-dimensional clues, and the three-dimensional clues can be extracted from a single or a small number of two-dimensional images without relying on the all-around shooting of the camera. The inventors have further realized that the three-dimensional clues can be converted into position constraints in a three-dimensional space, and then the three-dimensional space point cloud can be recovered from the two-dimensional image in combination with the position constraints. In this way, the three-dimensional space point cloud can provide key information for the three-dimensional reconstruction process without relying on the all-around shooting of the camera, and even in the case of limited shooting angle of the camera, the three-dimensional reconstruction process can also provide basic clues. In order to further improve the accuracy of three-dimensional reconstruction, the inventors have also realized that the three-dimensional space point cloud can be clustered according to the spatial correlation of the three-dimensional space point cloud. Thus, the point cloud belonging to the same target is clustered into a cluster, and the spatial range of the target can be accurately divided whether the shooting angle of the camera is limited or not.

[0085] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0086] The present application provides a three-dimensional reconstruction method of a single target. Figure 1 The present application provides a three-dimensional reconstruction method of a single target. Figure One As shown in the figure, the three-dimensional reconstruction method of the single target comprises: Figure 1

[0087] S101, inputting the pre-shooting multi-angle image and the preset prompt word into a target visual segmentation model, and obtaining the pixel-level mask of the target to be reconstructed output by the target visual segmentation model based on the multi-angle image and the preset prompt word.

[0088] The target visual segmentation model can use a current mainstream large pre-training visual segmentation model, including but not limited to the Segment Anything Model (SAM), the Segment Everything Everywhere All at Once (SEEM) model, etc. The target visual segmentation model used in the present embodiment is not limited here.

[0089] In the present embodiment, the image containing the target to be reconstructed at different angles is shot by the target camera to obtain the multi-angle image. The multi-angle image refers to a set of multiple images with different angles, and each image contains the target to be reconstructed.​

[0090] It should be understood that Figure 2 A shooting scene diagram of the three-dimensional reconstruction method of a single object provided in an embodiment of the present application Figure One As shown in Figure 2 If the single object to be reconstructed is located in an open space, the target camera can shoot the single object to be reconstructed at any position, meaning that the target camera has no angle limitation when shooting. Figure 3 A shooting scene diagram of the three-dimensional reconstruction method of a single object provided in an embodiment of the present application Figure Two As shown in Figure 3 If the single object to be reconstructed is located in a closed space, the target camera can only shoot the single object to be reconstructed in a limited area, and there may be an angle limitation problem.

[0091] It should be noted that the present embodiment is not affected by the location (open space / closed space) of the single object to be reconstructed.

[0092] It is worth noting that the multi-view images shot by the target camera need to be input into the target visual segmentation model together with the preset prompt word. The preset prompt word is used to define the category to which the single object to be reconstructed belongs.

[0093] Explanatorily, the format of the preset prompt word is usually natural language description or example image. By inputting the preset prompt word into the target visual segmentation model, the essence is to help the target visual segmentation model focus on the single object of a specific category, i.e., the single object to be reconstructed, from the complex scene presented by the multi-view images through the constraint of the preset prompt word. For example, if the preset prompt word is "table", the target visual segmentation model can preferentially segment the area in the multi-view images that belongs to the "table" category, and ignore walls, floors, other furniture, and other irrelevant elements.

[0094] Specifically, the steps of processing the multi-view images and the preset prompt word by the target visual segmentation model to generate the pixel-level mask of the single object to be reconstructed are as follows:

[0095] Step a1, feature extraction is performed on the multi-view images to determine the visual features of the multi-view images, and the preset prompt word is converted into a semantic vector.

[0096] It should be understood that the multi-view images contain the appearance, position, and geometric relationship of various objects (not only the single object to be reconstructed) at different angles, and this information exists in the form of pixels, which cannot be directly understood by the target visual segmentation model. Therefore, feature extraction is performed on the multi-view images to ensure that the target visual segmentation model sees details, i.e., the visual information included in the multi-view images.

[0097] In addition, the preset prompt word is a literal expression (natural language) or an image description (example image) of human intention, which cannot be directly understood by the target visual segmentation model. Converting the preset prompt word into a semantic vector is actually mapping the preset prompt word to a space that can be calculated by the target visual segmentation model, thereby helping the target visual segmentation model understand the semantic requirement, i.e., finding what from the multi-view image.

[0098] Step a2, matching the visual features with the semantic vector to predict the probability of each pixel in the multi-view image belonging to the single target to be reconstructed.

[0099] Explanatorily, matching the visual features with the semantic vector is essentially calculating the similarity of the visual features of each pixel with the semantic vector, thereby predicting the probability of each pixel belonging to the single target to be reconstructed. The higher the similarity, the more likely the pixel belongs to the single target to be reconstructed.

[0100] It should be noted that the calculated similarity can be converted into a corresponding probability value through a decoder, an attention mechanism, etc. The probability value is usually between 0 and 1, and a probability value close to 1 indicates that the pixel is extremely likely to belong to the single target to be reconstructed, and a probability value close to 0 indicates that the pixel belongs to the background or other single targets.

[0101] Step a3, determining the pixels with a probability greater than or equal to a preset threshold as the single target to be reconstructed, and determining the pixels with a probability less than the preset threshold as the background.

[0102] It should be understood that by setting a preset threshold (such as 0.5), the region with a pixel probability greater than the preset threshold is determined as the single target to be reconstructed, and the region with a pixel probability less than the preset threshold is determined as the background, thereby realizing the separation of the single target to be reconstructed and the background.

[0103] Step a4, binarizing the single target to be reconstructed and the background to generate a pixel-level mask of the single target to be reconstructed.

[0104] Explanatorily, this step is to convert the probability prediction into explicit region labeling. Specifically, the pixels with a probability greater than or equal to a preset threshold are labeled as 1, indicating that they belong to the single target to be reconstructed, and the pixels with a probability less than the preset threshold are labeled as 0, indicating that they belong to the background.

[0105] It should be understood that the finally generated pixel-level mask of the single target to be reconstructed is consistent in size with the original multi-view image, but each pixel only contains two values of "0" or "1".

[0106] S102, mapping the pixel-level mask to a three-dimensional space point cloud; and performing clustering processing on the three-dimensional space point cloud based on a preset algorithm to determine the geometric center point and the three-dimensional bounding box of the single target to be reconstructed.

[0107] Since the pixel-level mask is essentially a region marking on a two-dimensional plane, it cannot directly reflect the three-dimensional properties of the monolithic target to be reconstructed, such as depth, volume, and stereoscopic contour. Therefore, by mapping the pixel-level mask to a three-dimensional point cloud, the monolithic target to be reconstructed is endowed with three-dimensional spatial properties, so that the monolithic target to be reconstructed is converted from a planar region in a two-dimensional image to a stereoscopic structure with real spatial characteristics.

[0108] It should be noted that mapping the pixel-level mask to a three-dimensional point cloud requires relying on the camera parameters of the target camera. The camera parameters include external parameters and internal parameters.

[0109] Specifically, a preset pose estimation algorithm is used to calculate the translation vector and rotation vector of the target camera during image shooting. The translation vector indicates the spatial position of the target camera, and the rotation vector indicates the orientation of the target camera. Then, the external parameters of the target camera are determined according to the translation vector and the rotation vector, wherein the external parameters indicate the pose of the target camera.

[0110] Illustratively, the pose estimation algorithm solves the translation vector and the rotation vector by analyzing the correspondence between the features in the multi-view image and the three-dimensional space. It is worth noting that the pose estimation algorithm of the present embodiment is used to calculate the pose of the target camera, not the pose of the monolithic target to be reconstructed.

[0111] The pose estimation algorithm of the present embodiment can be a method based on known three-dimensional structures, a motion recovery structure algorithm, or a visual odometry algorithm, which is not limited by the pose estimation algorithm used in the present embodiment. The method based on known three-dimensional structures, the motion recovery structure algorithm, and the visual odometry algorithm are all common existing algorithms, which will not be described here.

[0112] It should be noted that the external parameters of the target camera are determined according to the translation vector and the rotation vector. First, the rotation vector is converted to a rotation matrix using a related formula (such as the Rodrigues formula), and then the translation vector is directly used as the translation component. Finally, the rotation matrix and the translation component are combined into a homogeneous matrix, which is the external parameter.

[0113] The internal parameters of the target camera can be obtained through a camera calibration process, wherein the internal parameters indicate the projection relationship from the three-dimensional camera coordinate system to the two-dimensional image pixel coordinate system. After obtaining the external parameters and the internal parameters of the target camera, the pixel-level mask is inversely projected in combination with the external parameters and the internal parameters, so as to map the pixel-level mask to a three-dimensional point cloud.

[0114] It should be understood that the two-dimensional pixel coordinates included in the pixel-level mask can be inversely deduced into three-dimensional coordinate points through the external parameters and the internal parameters, so as to generate a three-dimensional space point cloud based on the three-dimensional coordinate points. The three-dimensional coordinate points refer to corresponding rays or possible three-dimensional coordinates of the pixel-level mask in the three-dimensional space. Specifically, the internal parameters are responsible for converting the two-dimensional pixel coordinates corresponding to the pixel-level mask into the ray direction in the three-dimensional camera coordinate system; the external parameters unify the rays to the real world coordinate system to realize the spatial alignment of the multi-view rays. Finally, the pixel-level mask is upgraded to a three-dimensional space point cloud through ray intersection calculation.

[0115] It should be noted that although the pixel-level mask has filtered most of the background, the inverse projection process may still introduce a small amount of background points, such as outliers of adjacent objects of the single target to be reconstructed. Therefore, in order to accurately separate the single target to be reconstructed from the three-dimensional space point cloud and eliminate noise interference, it is necessary to purify the three-dimensional space point cloud and separate the target.

[0116] Specifically, a preset algorithm is used to perform clustering processing on the three-dimensional space point cloud. The clustering processing gathers points with similar distances and belonging to the same target into a cluster by calculating the spatial distances between the discrete points included in the three-dimensional space point cloud, and filters out noise and outliers with too far distances.

[0117] In this embodiment, the preset algorithm used is an unsupervised spatial clustering algorithm. Specifically, in combination with the spatial distribution characteristics (such as the distance, density, connectivity, etc. between points) of the three-dimensional space point cloud, the unsupervised spatial clustering algorithm is used to divide the three-dimensional space point cloud into multiple point cloud subsets (clusters). Each point cloud subset corresponds to a single target.

[0118] Explanatorily, the core of the unsupervised spatial clustering algorithm is to utilize the inherent spatial correlation of the three-dimensional space point cloud, including but not limited to distance features, density features, connectivity features, etc. The distance features refer to that the points belonging to the same single target are usually close in space. The density features refer to that the point cloud on the surface of the single target to be reconstructed is usually dense, while the background or outliers are sparse. The connectivity features refer to that the points belonging to the same single target form a connected region through continuous spatial distribution.

[0119] It should be noted that the unsupervised spatial clustering algorithm can use existing algorithms suitable for three-dimensional space point clouds, including but not limited to Euclidean clustering, region growing clustering, and density-based noisy spatial clustering. These clustering algorithms are common existing algorithms, which will not be described here.

[0120] It can be understood that the three-dimensional space point cloud after clustering processing includes a plurality of point cloud subsets, but each point cloud subset still includes a series of discrete points. In order to make the point cloud subset corresponding to the monomer target to be reconstructed have a clear physical meaning, it is also necessary to: first, from the plurality of point cloud subsets included in the three-dimensional space point cloud, determine a target point cloud subset containing the monomer target to be reconstructed. Then, according to the three-dimensional coordinates of the target point cloud subset, the geometric center point and the three-dimensional bounding box of the monomer target to be reconstructed are calculated.

[0121] Among them, the calculation of the geometric center point is: the average value of the three-dimensional coordinates of all points in the target point cloud subset. The three-dimensional bounding box refers to the smallest three-dimensional rectangular body that can completely wrap the target point cloud subset, and the calculation of the three-dimensional bounding box depends on the three-dimensional coordinate extreme value of the target point cloud subset.

[0122] It should be understood that the calculation of the geometric center point and the three-dimensional bounding box is essentially to convert the low-level coordinate information of the discrete point cloud into high-level physical attributes (such as position, size, space occupation, etc.). These physical attributes directly correspond to the basic characteristics of objects in the real world and can be understood and applied.

[0123] S103, input the geometric center point and the three-dimensional bounding box into the target three-dimensional reconstruction model to obtain a three-dimensional model of the monomer target to be reconstructed output by the target three-dimensional reconstruction model.

[0124] Specifically, after inputting the geometric center point and the three-dimensional bounding box into the target three-dimensional reconstruction model, the target three-dimensional reconstruction model will use its own internal preset sampling strategy and preset spatial constraints, and combine the geometric center point and the three-dimensional bounding box to perform three-dimensional modeling on the monomer target to be reconstructed.

[0125] Among them, the preset sampling strategy is the rule for the target three-dimensional reconstruction model to select sampling points in three-dimensional space, and the core purpose is to capture the most critical three-dimensional geometric information with the least computing resources. The preset spatial constraint is a physical or logical limit imposed by the target three-dimensional reconstruction model on the generated three-dimensional geometric structure, and the core purpose is to ensure that the reconstruction result conforms to the laws of the real world.

[0126] Explanatorily, the three-dimensional bounding box defines the maximum spatial range in which the monomer target to be reconstructed may exist, and the target three-dimensional reconstruction model does not need to blindly learn in the entire space, but only needs to focus on the radiation field within the three-dimensional bounding box. The geometric center point marks the core position of the monomer target to be reconstructed, prompting the target three-dimensional reconstruction model that the high-density area (the monomer target to be reconstructed) within the three-dimensional bounding box is most likely to be distributed around the geometric center point, further narrowing the range of the key area.

[0127] It can be seen that the preset sampling strategy and the preset spatial constraint provide a general modeling logic, but lack spatial information of the monomer target to be reconstructed. The geometric center point and the three-dimensional bounding box provide prior information for the target three-dimensional reconstruction model, and can accurately frame the modeling range.

[0128] It should be noted that the process of the target three-dimensional reconstruction model for three-dimensional modeling of the monomer target to be reconstructed is as follows: based on the geometric center point and the three-dimensional bounding box, the reconstruction region of the target three-dimensional reconstruction model is determined. Then the target three-dimensional reconstruction model samples in the reconstruction region to generate corresponding sampling points. Further, according to the sampling points, the monomer target to be reconstructed is three-dimensionally modeled in the reconstruction region, thereby generating and outputting a three-dimensional model of the monomer target to be reconstructed. The three-dimensional model is a digital representation of the three-dimensional spatial structure of the monomer target to be reconstructed.

[0129] It should be understood that the preset sampling strategy acts on the sampling stage in the above process, and the preset spatial constraint acts on the three-dimensional modeling stage in the above process.

[0130] In one possible implementation, the three-dimensional reconstruction method used by the target three-dimensional reconstruction model of the embodiment is neural volume rendering. In neural volume rendering, the sampling stage depends on the reconstruction region and the preset sampling strategy to select representative points (sampling points) in the three-dimensional space, providing accurate spatial coordinate samples for the three-dimensional modeling process. The three-dimensional modeling stage depends on the preset spatial constraint and the sampling points to learn the radiation field (density and color distribution) based on the sampling points, and ensure that the density distribution conforms to the physical properties of the monomer target to be reconstructed through the preset spatial constraint.

[0131] Specifically, the process of the sampling stage is as follows: for each pixel of the pixel-level mask, a three-dimensional ray passing through the pixel is emitted from the optical center of the target camera, and then sampling is performed only on the line segment where the three-dimensional ray intersects the three-dimensional bounding box. Further, a series of sampling point coordinates and corresponding viewing directions (used for predicting color) located in the reconstruction region are generated. Among them, more sampling points are allocated to the area near the geometric center point on the line segment, and fewer sampling points are allocated to the edge area.

[0132] The process of the three-dimensional reconstruction stage is as follows: the coordinates and viewing directions of the sampling points are taken as inputs to predict the density and color of the point. Then the density and color of the point are combined into an image pixel value through a related volume rendering formula, and the image pixel value is compared with the pixel-level mask to calculate the loss. Thus, the network parameters are optimized in reverse, so that the radiation field gradually fits the real geometry and appearance of the monomer target to be reconstructed.

[0133] In summary, the three-dimensional reconstruction method of the monomer target of the embodiment is summarized. Figure 4 The flowchart of the three-dimensional reconstruction method of the monomer target provided by the embodiment of the present application Figure Two For example,Figure 4 As shown, the specific steps of the three-dimensional reconstruction method of the monomer target of the embodiment are as follows:

[0134] S401, using a target camera to shoot images containing a monomer target to be reconstructed at different angles of view, and generating multi-view images.

[0135] S402, inputting the multi-view images and the preset prompt words into a target visual segmentation model.

[0136] S403, the target visual segmentation model outputs a pixel-level mask of the monomer target to be reconstructed based on the multi-view images and the preset prompt words.

[0137] S404, using a pose estimation algorithm to determine the external parameters of the target camera, and obtaining the internal parameters of the target camera.

[0138] S405, forming the camera parameters of the target camera based on the external parameters and the internal parameters.

[0139] It should be understood that S402-S403 and S404-S405 do not have a strict execution sequence, and can be executed simultaneously.

[0140] S406, based on the camera parameters, performing inverse projection processing on the pixel-level mask to map the pixel-level mask to a three-dimensional space point cloud.

[0141] It should be noted that in one possible design, if the pixel-level mask has depth information, the camera parameters and the depth information need to be combined to perform inverse projection processing on the pixel-level mask. If the pixel-level mask does not have depth information, the pixel-level mask is inversely projected according to the camera parameters. The depth information refers to the distance from the object point in the real world corresponding to the pixel point in the pixel-level mask to the target camera.

[0142] S407, using an unsupervised spatial clustering algorithm to cluster the three-dimensional space point cloud.

[0143] S408, determining the geometric center point and the three-dimensional bounding box of the monomer target to be reconstructed.

[0144] S409, inputting the geometric center point and the three-dimensional bounding box into a target three-dimensional reconstruction model.

[0145] S410, the target three-dimensional reconstruction model performs three-dimensional modeling on the monomer target to be reconstructed, and outputs a three-dimensional model of the monomer target to be reconstructed.

[0146] The three-dimensional reconstruction method of the monomer target provided in the application inputs the pre-shot multi-view image and the preset prompt word into a target visual segmentation model. The target visual segmentation model extracts features from the multi-view image to obtain visual features of the multi-view image, and simultaneously converts the preset prompt word into a semantic vector. Then the visual features and the semantic vector are matched to predict the probability of each pixel in the multi-view image belonging to the monomer target to be reconstructed, and the pixels with a probability greater than or equal to a preset threshold are determined as the monomer target to be reconstructed, and the pixels with a probability less than the preset threshold are determined as the background. Next, the monomer target to be reconstructed and the background are binarized and labeled to generate a pixel-level mask of the monomer target to be reconstructed. Next, in addition to obtaining the internal parameters of the target camera, a pose estimation algorithm is used to calculate the translation vector and the rotation vector of the target camera during the image shooting process, and the external parameters of the target camera are generated according to the translation vector and the rotation vector. The pixel-level mask is subjected to inverse projection processing in combination with the external parameters and the internal parameters of the target camera, so as to map the pixel-level mask to a three-dimensional space point cloud, realizing the conversion of the two-dimensional image information to the potential position constraint in the three-dimensional space. Even in the case of limited view angle, it can also provide a basic clue for the positioning of the monomer target to be reconstructed. Further, an unsupervised spatial clustering algorithm is used to divide the three-dimensional space point cloud into multiple point cloud subsets, and then a target point cloud subset containing the monomer target to be reconstructed is determined from the multiple point cloud subsets. According to the three-dimensional coordinates of the target point cloud subset, the geometric center point and the three-dimensional bounding box of the monomer target to be reconstructed are calculated. The clustering processing does not depend on the integrity of the view angle, and by mining the spatial correlation of the three-dimensional space point cloud, the point clouds belonging to the same monomer target are clustered into a cluster, so that the spatial range of the monomer target to be reconstructed can be accurately divided even if the view angle is not complete. Finally, the geometric center point and the three-dimensional bounding box are input into a target three-dimensional reconstruction model, and the target three-dimensional reconstruction model uses a preset sampling strategy and a preset spatial constraint to perform three-dimensional modeling on the monomer target to be reconstructed to generate a three-dimensional model of the monomer target to be reconstructed. As can be seen, whether the shooting view angle of the target camera is limited or not, the geometric center point and the three-dimensional bounding box of the monomer target to be reconstructed can be accurately inferred. On the basis of the accurate geometric center point and the three-dimensional bounding box, the three-dimensional model of the monomer target to be reconstructed output by the target three-dimensional reconstruction model is also accurate and reliable.

[0147] Figure 5 The structure diagram of the three-dimensional reconstruction device of the monomer target provided in the embodiment of the application is shown in Figure 5 as shown, the three-dimensional reconstruction device of the monomer target 500 comprises an input module 501, an acquisition module 502, a mapping module 503 and a processing module 504.

[0148] The input module 501 is configured to input the pre-shot multi-view image and the preset prompt word into the target visual segmentation model.

[0149] The acquisition module 502 is configured to acquire a pixel-level mask of a target monomer object output by a target visual segmentation model based on multi-view images and a preset prompt word, where the multi-view images represent a set of images with different perspectives, and each image included in the multi-view images contains the target monomer object; and the preset prompt word is used to define a category to which the target monomer object belongs.

[0150] The mapping module 503 is configured to map the pixel-level mask to a three-dimensional space point cloud.

[0151] The processing module 504 is configured to perform clustering processing on the three-dimensional space point cloud based on a preset algorithm, to determine a geometric center point and a three-dimensional bounding box of the target monomer object.

[0152] The input module 501 is further configured to input the geometric center point and the three-dimensional bounding box to a target three-dimensional reconstruction model, to obtain a three-dimensional model of the target monomer object output by the target three-dimensional reconstruction model, where the three-dimensional model represents a digital representation of a three-dimensional space structure of the target monomer object.

[0153] In a possible design, the mapping module 503 further includes a calculation module 505 and a determination module 506.

[0154] The calculation module 505 is configured to calculate a translation vector and a rotation vector of a target camera in an image shooting process by using a preset pose estimation algorithm.

[0155] The determination module 506 is configured to determine external parameters of the target camera based on the translation vector and the rotation vector, where the target camera is used to shoot images containing the target monomer object from different perspectives; the translation vector indicates a spatial position of the target camera, the rotation vector indicates an orientation of the target camera, and the external parameters indicate a pose of the target camera.

[0156] The acquisition module 502 is further configured to acquire internal parameters of the target camera.

[0157] The processing module 504 is further configured to perform inverse projection processing on the pixel-level mask based on the external parameters and the internal parameters, to map the pixel-level mask to the three-dimensional space point cloud, where the internal parameters indicate a projection relationship from a three-dimensional camera coordinate system to a two-dimensional image pixel coordinate system.

[0158] In a possible design, the determination module 506 is further configured to determine two-dimensional coordinates of pixels included in the pixel-level mask.

[0159] The processing module 504 further includes a conversion module 507 and a generation module 508.

[0160] The conversion module 507 is configured to convert the two-dimensional coordinates to three-dimensional coordinate points based on the external parameters and the internal parameters.

[0161] The generating module 508 is configured to generate a three-dimensional space point cloud based on the three-dimensional coordinate points.

[0162] In a possible design, the preset algorithm is an unsupervised spatial clustering algorithm.

[0163] The processing module 504 further includes a dividing module 509 configured to divide the three-dimensional space point cloud into a plurality of point cloud subsets based on the unsupervised spatial clustering algorithm in combination with a spatial distribution feature of the three-dimensional space point cloud.

[0164] The determining module 506 is further configured to determine, from the plurality of point cloud subsets, a target point cloud subset containing the single-body target to be reconstructed.

[0165] The calculating module 505 is further configured to calculate a geometric center point and a three-dimensional bounding box of the single-body target to be reconstructed based on the three-dimensional coordinates of the target point cloud subset.

[0166] In a possible design, the inputting module 501 is further configured to input the geometric center point and the three-dimensional bounding box to a target three-dimensional reconstruction model.

[0167] The processing module 504 is further configured to perform three-dimensional modeling on the single-body target to be reconstructed based on a preset sampling strategy and a preset spatial constraint inside the target three-dimensional reconstruction model in combination with the geometric center point and the three-dimensional bounding box.

[0168] The obtaining module 502 is further configured to obtain a three-dimensional model of the single-body target to be reconstructed output by the target three-dimensional reconstruction model.

[0169] In a possible design, the determining module 506 is further configured to determine a reconstruction region of the target three-dimensional reconstruction model based on the geometric center point and the three-dimensional bounding box.

[0170] The processing module 504 further includes a sampling module 510 configured to perform sampling in the reconstruction region based on the target three-dimensional reconstruction model to generate a corresponding sampling point.

[0171] The processing module 504 is further configured to perform three-dimensional modeling on the single-body target to be reconstructed in the reconstruction region based on the sampling point.

[0172] In a possible design, the obtaining module 502 further includes an extracting module 511, a matching module 512, and a marking module 513.

[0173] The extracting module 511 is configured to perform feature extraction on the multi-view images based on the target visual segmentation model to determine visual features of the multi-view images.

[0174] The converting module 507 is further configured to convert the preset prompt word into a semantic vector.

[0175] The matching module 512 is configured to match the visual features with the semantic vectors, and predict a probability that each pixel in the multi-view image belongs to the single object to be reconstructed.

[0176] The determining module 506 is further configured to determine a pixel whose probability is greater than or equal to a preset threshold as the single object to be reconstructed, and determine a pixel whose probability is less than the preset threshold as the background.

[0177] The marking module 513 is configured to perform binary marking on the single object to be reconstructed and the background, and generate a pixel-level mask of the single object to be reconstructed.

[0178] The three-dimensional reconstruction device for a single object provided by the embodiments of the present application can be used to execute the three-dimensional reconstruction method for a single object in any of the above embodiments, and has similar implementation principles and technical effects, which will not be described here in detail.

[0179] It should be noted that the division of each module of the above device is only a logical functional division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. Moreover, all of the modules can be implemented in the form of software through a processing element; all of the modules can also be implemented in the form of hardware; some modules can be implemented in the form of software through a processing element, and some modules can be implemented in the form of hardware. In addition, all or part of the modules can be integrated together, or can be independently implemented. The processing element herein can be an integrated circuit having a signal processing capability. In the implementation process, each step of the above method or each module can be completed by an integrated logic circuit of hardware or an instruction in the form of software in the processing element.

[0180] Figure 6 The structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, the electronic device can include a transceiver 61, a processor 62, and a memory 63. Figure 6

[0181] The processor 62 executes the computer execution instructions stored in the memory, so that the processor 62 executes the schemes in the above embodiments. The processor 62 can be a general-purpose processor, including a central processing unit CPU, a network processor NP, etc.; and can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0182] The memory 63 is connected with the processor 62 through a system bus and completes mutual communication, and the memory 63 is used to store computer program instructions.

[0183] The transceiver 61 can be used for communication interaction with other devices.​

[0184] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The system bus can be divided into an address bus, a data bus, a control bus, or the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries and read-only libraries). The memory can include random access memory (RAM), and can also include non-volatile memory.

[0185] The electronic device provided by the embodiments of the present application can be used to execute the method provided by any of the above embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0186] The embodiments of the present application further provide a computer readable storage medium, which stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method provided by any of the above embodiments.

[0187] The embodiments of the present application further provide a computer program product, which includes a computer program stored in a computer readable storage medium, at least one processor can read the computer program from the computer readable storage medium, and when the at least one processor executes the computer program, the method provided by any of the above embodiments can be implemented.

[0188] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0189] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to implement the embodiments of the present application.

[0190] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The above-mentioned modules can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0191] The integrated modules realized in the form of software functional modules can be stored in a computer readable storage medium. The software functional modules stored in a storage medium include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method of each embodiment of the present application.

[0192] It should be understood that the above-mentioned processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0193] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, for example at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0194] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0195] The storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0196] An exemplary storage medium is coupled to the processor so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can be located in a remote terminal or a host computer.

[0197] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program, when executed, executes steps including the above-mentioned method embodiments; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of three-dimensional reconstruction of a single object, characterized in that, The method comprises: inputting a pre-shot multi-view image and a preset prompt word into a target visual segmentation model, and obtaining a pixel-level mask of a to-be-reconstructed single object output by the target visual segmentation model based on the multi-view image and the preset prompt word; wherein the multi-view image represents a set of multiple images with different viewing angles, and each image included in the multi-view image contains the to-be-reconstructed single object; the preset prompt word is used to define the category of the to-be-reconstructed single object; mapping the pixel-level mask to a three-dimensional space point cloud; and performing clustering processing on the three-dimensional space point cloud based on a preset algorithm to determine a geometric center point and a three-dimensional bounding box of the to-be-reconstructed single object; inputting the geometric center point and the three-dimensional bounding box into a target three-dimensional reconstruction model to obtain a three-dimensional model of the to-be-reconstructed single object output by the target three-dimensional reconstruction model; wherein the three-dimensional model represents a digital representation of the three-dimensional spatial structure of the to-be-reconstructed single object.

2. The method of claim 1, wherein, The method comprises: using a preset pose estimation algorithm to calculate a translation vector and a rotation vector of a target camera in an image shooting process; determining external parameters of the target camera based on the translation vector and the rotation vector; wherein the target camera is used to shoot images containing the to-be-reconstructed single object at different viewing angles; the translation vector indicates the spatial position of the target camera, the rotation vector indicates the orientation of the target camera, and the external parameters indicate the pose of the target camera; obtaining internal parameters of the target camera; and performing inverse projection processing on the pixel-level mask based on the external parameters and the internal parameters to map the pixel-level mask to a three-dimensional space point cloud; wherein the internal parameters indicate the projection relationship from a three-dimensional camera coordinate system to a two-dimensional image pixel coordinate system.

3. The method of claim 2, wherein, The method comprises: determining the two-dimensional coordinates of the pixels included in the pixel-level mask; converting the two-dimensional coordinates to three-dimensional coordinate points based on the external parameters and the internal parameters; and generating a three-dimensional space point cloud based on the three-dimensional coordinate points.

4. The method of claim 1, wherein, The preset algorithm is an unsupervised spatial clustering algorithm; The method comprises: combining the spatial distribution characteristics of the three-dimensional space point cloud, dividing the three-dimensional space point cloud into multiple point cloud subsets based on the unsupervised spatial clustering algorithm; from the multiple point cloud subsets, determining a target point cloud subset containing the to-be-reconstructed single object; and based on the three-dimensional coordinates of the target point cloud subset, calculating the geometric center point and the three-dimensional bounding box of the to-be-reconstructed single object.

5. The method of claim 1, wherein, The method comprises: inputting the geometric center point and the three-dimensional bounding box into a target three-dimensional reconstruction model; based on the geometric center point and the three-dimensional bounding box, performing three-dimensional modeling on the single object to be reconstructed based on a preset sampling strategy and a preset spatial constraint inside the target three-dimensional reconstruction model; obtaining a three-dimensional model of the single object to be reconstructed output by the target three-dimensional reconstruction model.

6. The method of claim 5, wherein, based on the geometric center point and the three-dimensional bounding box, performing three-dimensional modeling on the single object to be reconstructed based on a preset sampling strategy and a preset spatial constraint inside the target three-dimensional reconstruction model, comprising: based on the geometric center point and the three-dimensional bounding box, determining a reconstruction area of the target three-dimensional reconstruction model; based on the target three-dimensional reconstruction model, sampling in the reconstruction area to generate corresponding sampling points; based on the sampling points, performing three-dimensional modeling on the single object to be reconstructed in the reconstruction area.

7. The method according to any one of claims 1 to 6, characterized in that, obtaining a pixel-level mask of the single object to be reconstructed output by the target visual segmentation model based on the multi-view images and the preset prompt word, comprising: based on the target visual segmentation model, performing feature extraction on the multi-view images to determine the visual features of the multi-view images; and converting the preset prompt word into a semantic vector; matching the visual features and the semantic vector to predict the probability of each pixel in the multi-view images belonging to the single object to be reconstructed; determining pixels with a probability greater than or equal to a preset threshold as the single object to be reconstructed, and determining pixels with a probability less than the preset threshold as background; performing binary labeling on the single object to be reconstructed and the background to generate a pixel-level mask of the single object to be reconstructed.

8. A device for three-dimensional reconstruction of a single object, characterized in that comprising: an input module configured to input pre-shot multi-view images and a preset prompt word into a target visual segmentation model; an obtaining module configured to obtain a pixel-level mask of a single object to be reconstructed output by the target visual segmentation model based on the multi-view images and the preset prompt word; wherein the multi-view images represent a set of multiple images with different angles, and each image included in the multi-view images contains the single object to be reconstructed; the preset prompt word is used to define the category to which the single object to be reconstructed belongs; a mapping module configured to map the pixel-level mask to a three-dimensional space point cloud; a processing module configured to perform clustering processing on the three-dimensional space point cloud based on a preset algorithm to determine a geometric center point and a three-dimensional bounding box of the single object to be reconstructed; the input module is further configured to input the geometric center point and the three-dimensional bounding box into a target three-dimensional reconstruction model to obtain a three-dimensional model of the single object to be reconstructed output by the target three-dimensional reconstruction model; wherein the three-dimensional model represents a digital representation of the three-dimensional spatial structure of the single object to be reconstructed.

9. An electronic device, comprising: comprising: a processor, and a memory in communication with the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the three-dimensional reconstruction method of the single object according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are used for realizing the three-dimensional reconstruction method of the single object according to any one of claims 1 to 7 when executed by the processor.

Images