Object bounding box determination method, three-dimensional reconstruction method and device, equipment and medium

By adjusting the camera position at multiple perspectives to estimate more accurate object bounding boxes, the problem of inaccurate bounding boxes in the prior art is solved, and the reconstruction accuracy and detailed completeness of the three-dimensional model are improved.

CN116342829BActive Publication Date: 2024-12-03NETEASE (HANGZHOU) NETWORK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310230722.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-03
Publication Date
2024-12-03
Estimated Expiration
2043-03-03

AI Technical Summary

Technical Problem

In the prior art, the implicit surface reconstruction algorithm based on neural rendering is not accurate enough in the estimation of object bounding box, resulting in limited reconstruction accuracy and detailed completeness of three-dimensional model.

Method used

By obtaining the 2D image of the target object at multiple perspectives, determining the initial camera parameters and 3D point clouds, re-adjust the camera position to estimate more accurate object bounding boxes, and using them for 3D reconstruction, improving the estimation accuracy of bounding boxes degree and reconstruction accuracy.

Benefits of technology

The bounding box estimation is achieved that is tighter and closer to the actual surface of the object, which improves network training accuracy and three-dimensional reconstruction accuracy, and enhances the integrity of model details.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116342829B_ABST
    Figure CN116342829B_ABST
Patent Text Reader

Abstract

The present application provides a method for determining an object bounding box, a three-dimensional reconstruction method and apparatus, equipment and medium, and relates to the field of computer vision technology. The method includes: obtaining multiple two-dimensional images of a target object at multiple perspectives; determining the initial camera parameters and three-dimensional point clouds at multiple perspectives respectively according to the multiple two-dimensional images, wherein the initial camera parameters include: the initial camera position; determining the target camera position at each perspective according to the initial camera position and three-dimensional point cloud at each perspective; determining the information of the bounding box of the target object according to the target camera position at multiple perspectives; and constructing the bounding box of the target object in three-dimensional space according to the information of the bounding box. The embodiments of the present application can improve the reconstruction accuracy of the three-dimensional model and the completeness of the model details.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method for determining an object bounding box, a three-dimensional reconstruction method and apparatus, equipment and medium. Background Art

[0002] Multi-view 3D reconstruction technology can collect 2D images from multiple viewpoints and reconstruct a 3D model. 3D reconstruction technology plays an important role in 3D games, virtual reality, augmented reality and other fields.

[0003] In order to avoid large holes on the surface of the reconstructed 3D model and ensure the complete continuity of its surface, the application of implicit surface reconstruction algorithms based on neural rendering is increasing. At present, whether it is the training process of the implicit surface reconstruction algorithm based on neural rendering or the process of 3D reconstruction based on the implicit surface reconstruction algorithm, the object bounding box estimated by the relevant technology is usually not accurate enough and may deviate from the actual object surface. Therefore, it is likely that the algorithm trained based on the object bounding box is not accurate enough, and the reconstruction accuracy of the 3D model is limited, and the model details are not complete. Summary of the invention

[0004] In view of this, embodiments of the present application provide a method for determining an object bounding box, a three-dimensional reconstruction method and apparatus, a device, and a medium to improve the reconstruction accuracy of a three-dimensional model and the completeness of model details.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Acquire multiple two-dimensional images of the target object at multiple viewing angles;

[0007] Determine initial camera parameters and three-dimensional point clouds under the multiple viewing angles respectively according to the multiple two-dimensional images, wherein the initial camera parameters include: an initial camera position;

[0008] Determine the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle;

[0009] Determining information of a bounding box of the target object according to the target camera positions under the multiple viewing angles;

[0010] A bounding box of the target object in three-dimensional space is constructed according to the information of the bounding box.

[0011] In a second aspect, an embodiment of the present application further provides a three-dimensional reconstruction method, the method comprising:

[0012] Acquire a bounding box of the target object; wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by any object bounding box determination method described in the first aspect above;

[0013] Inputting a plurality of sampling points within the bounding box into a pre-trained surface distance extraction network to obtain a target symbol distance parameter;

[0014] According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction on the target object to obtain three-dimensional surface data of the target object.

[0015] In a third aspect, an embodiment of the present application further provides a device for determining an object bounding box, comprising:

[0016] A first acquisition module is used to acquire multiple two-dimensional images of the target object at multiple viewing angles;

[0017] A determination module, configured to determine initial camera parameters and three-dimensional point clouds at the multiple viewing angles respectively according to the multiple two-dimensional images, wherein the initial camera parameters include: an initial camera position; determining a target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle; and determining information of a bounding box of the target object according to the target camera position at the multiple viewing angles;

[0018] A construction module is used to construct a bounding box of the target object in three-dimensional space according to the information of the bounding box.

[0019] In a fourth aspect, the present application also provides a three-dimensional reconstruction device, including:

[0020] A second acquisition module, configured to acquire a bounding box of a target object; wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by any object bounding box determination method described in the first aspect;

[0021] An input module, used for inputting a plurality of sampling points in the bounding box into a pre-trained surface distance extraction network to obtain a target symbol distance parameter;

[0022] The reconstruction module is used to perform three-dimensional reconstruction on the target object according to the target symbol distance parameter by using a preset surface extraction algorithm to obtain three-dimensional surface data of the target object.

[0023] In a fifth aspect, an embodiment of the present application further provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the object bounding box determination method described in any one of the first aspect or the three-dimensional reconstruction method described in the second aspect.

[0024] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the object bounding box determination method described in any one of the first aspect or the three-dimensional reconstruction method described in the second aspect are executed.

[0025] The object bounding box determination method, three-dimensional reconstruction method and apparatus, device and medium provided in the embodiments of the present application can obtain multiple two-dimensional images of the target object at multiple perspectives, first determine the initial camera parameters and three-dimensional point clouds at multiple perspectives respectively according to the multiple two-dimensional images, wherein the initial camera parameters include: an initial camera position, and then determine the target camera position at each perspective according to the initial camera position and the three-dimensional point cloud at each perspective, and then determine the bounding box information of the target object according to the target camera position at multiple perspectives, and construct the bounding box of the target object in three-dimensional space according to the bounding box information. Because in the method provided in the embodiment of the present application, when the initial camera parameters and three-dimensional point clouds at multiple perspectives are obtained based on multiple two-dimensional images, the target camera position at each perspective is re-determined according to the initial camera position and the three-dimensional point cloud at each perspective, thereby realizing the translation of the camera position at each perspective, and then estimating the object bounding box based on the target camera position at multiple perspectives, it not only realizes the automatic estimation of the bounding box of the target object, but also makes the estimated bounding box of the target object more compact and closer to the actual object surface, thereby improving the estimation accuracy of the object bounding box, thereby improving the training accuracy of the network training based on the bounding box, and the three-dimensional reconstruction accuracy of the target object, and further improving the detail completeness of the model.

[0026] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A flowchart of a method for determining an object bounding box provided in an embodiment of the present application;

[0029] Figure 2 A flow chart of a method for determining a target camera position at each viewing angle in a data processing method provided in an embodiment of the present application;

[0030] Figure 3 A comparison diagram of the bounding box of the target object provided by the embodiment of the present application and the bounding box of the object in the traditional solution;

[0031] Figure 4 A flow chart of a method for determining a bounding box of a target object in a data processing method provided in an embodiment of the present application;

[0032] Figure 5 A flow chart of a method for performing volume density conversion based on a bounding box of a target object provided in an embodiment of the present application;

[0033] Figure 6 A flow chart of another method for performing volume density conversion based on a bounding box of a target object provided in an embodiment of the present application;

[0034] Figure 7 A flow chart of a method for training a surface distance extraction network provided in an embodiment of the present application;

[0035] Figure 8 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of the structure of a device for determining an object bounding box provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0038] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0040] The object bounding box determination method in one of the embodiments of the present disclosure can be applied to the process of training the algorithm model used in the three-dimensional reconstruction process, and can also be applied to the process of three-dimensional reconstruction of the object. The three-dimensional reconstruction involved in the embodiments of the present application can be applied to any application scenario involving virtual three-dimensional models, such as three-dimensional game scenes, virtual scenes based on virtual reality, virtual scenes based on augmented reality, etc. In the specific execution process, the object bounding box determination method and the three-dimensional reconstruction method can be run together on the same device body, or they can be run on different device bodies respectively. Whether it is the object bounding box determination method or the three-dimensional reconstruction method, the device body of the running method can be a local terminal device or a server. When the method runs on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0041] In order to facilitate a better understanding of the object bounding box determination method and the three-dimensional reconstruction method provided in the embodiments of the present application, the technical terms involved in the following embodiments of the present application are first explained as follows.

[0042] Technical term explanation:

[0043] SDF: Signed Distance Function (SDF), which is an implicit expression of the surface of a 3D model, can determine the distance from a 3D point to the surface of an object in a finite area in 3D space and define the sign of the distance at the same time. For example, if the point is inside the object, it is positive; if the point is outside the object, it is negative; if the point is on the surface of the object, it is 0.

[0044] MLP: Multilayer Perceptron (MLP), a type of neural network architecture.

[0045] BBox: Bounding Box (BBox), refers to the minimum bounding box surrounding an object in three-dimensional (3D) space.

[0046] SFM: Structure From Motion (SFM) is a technique for estimating three-dimensional structure from a series of two-dimensional images containing visual motion information. SFM can be used to obtain camera parameters and sparse point clouds for each perspective.

[0047] Logistic distribution: Logistic distribution, also known as probability distribution.

[0048] Marching Cubes: Surface extraction algorithm, also known as surface rendering algorithm, is an algorithm used to extract isosurfaces in volume data. The core concept is to determine which edges of voxels intersect with isosurfaces by comparing vertices with user-specified thresholds after setting the 3D space and vertex scalar values, create triangular patches, and connect all cube faces on the isosurface boundary to obtain a surface.

[0049] In order to clearly describe the three-dimensional reconstruction method provided in the embodiment of the present application, the object bounding box method provided in the embodiment of the present application is first explained and illustrated by way of examples in combination with multiple embodiments in combination with multiple drawings as follows, so as to clearly explain the object bounding box used in the three-dimensional reconstruction method and the object bounding box used in the training process of the algorithm model involved in the three-dimensional reconstruction method.

[0050] Figure 1 A flowchart of a method for determining an object bounding box provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the object bounding box determination method may include:

[0051] S101, acquiring multiple two-dimensional images of a target object at multiple viewing angles.

[0052] For example, a plurality of two-dimensional images may be obtained by photographing a target object in a real environment with a preset camera at multiple viewing angles, and the plurality of two-dimensional images correspond to the multiple viewing angles, respectively.

[0053] S102: Determine initial camera parameters and three-dimensional point clouds under multiple viewing angles according to the multiple two-dimensional images.

[0054] In a possible implementation, a preset pose estimation algorithm may be used to perform joint pose estimation on multiple two-dimensional images to obtain three-dimensional point clouds at multiple perspectives and initial camera parameters at multiple perspectives, wherein the three-dimensional point cloud at each perspective may be, for example, a sparse point cloud. The preset pose estimation algorithm may be, for example, an SFM network. The initial camera parameters at each perspective may include: camera pose information at that perspective, which may at least include: an initial camera position.

[0055] S103: Determine the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle.

[0056] In a possible implementation, the initial camera parameters at each viewing angle may be adjusted by translation according to the three-dimensional point cloud at each viewing angle to obtain the target camera parameters at each viewing angle, so as to translate the camera position at each viewing angle closer to the three-dimensional point cloud of the target object at each viewing angle.

[0057] S104: Determine information of a bounding box of the target object according to the target camera positions under multiple viewing angles.

[0058] In actual application scenarios, when the target camera positions under multiple viewing angles are obtained, the bounding box of the target object can be estimated according to the target camera positions under multiple viewing angles to obtain the bounding box information of the target object.

[0059] S105 . Construct a bounding box of the target object in three-dimensional space according to the bounding box information.

[0060] When the bounding box information of the target object is obtained by adopting the above S104, a three-dimensional bounding box can be constructed in three-dimensional space based on the bounding box information to obtain the bounding box of the target object. Accordingly, the bounding box of the target object can be, for example, a three-dimensional object bounding box.

[0061] In the method provided in the embodiment of the present application, when initial camera parameters and three-dimensional point clouds under multiple viewing angles are obtained based on multiple two-dimensional images, the target camera position under each viewing angle is re-determined according to the initial camera position and the three-dimensional point cloud under each viewing angle, thereby realizing the translation of the camera position under each viewing angle, and then estimating the object bounding box based on the target camera position under multiple viewing angles, which not only realizes the automatic estimation of the bounding box of the target object, but also makes the estimated bounding box of the target object more compact and closer to the actual object surface, thereby improving the estimation accuracy of the object bounding box, thereby improving the training accuracy of network training based on the bounding box and the three-dimensional reconstruction accuracy of the target object, and then the completeness of the model details.

[0062] Secondly, since the method provided in this embodiment can automatically calculate a more compact bounding box for the target object and estimate the object bounding box more accurately, this method also reduces the application complexity of network training and three-dimensional reconstruction process.

[0063] For the translation update of the target camera position, the present application embodiment also provides some possible implementations, which are described below in conjunction with the accompanying drawings. The initial camera parameters shown above may also include at least: a camera rotation matrix. Figure 2 A flow chart of a method for determining the target camera position at each viewing angle in the data processing method provided in an embodiment of the present application, such as Figure 2 As shown, in S102 shown above, determining the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle may include:

[0064] S201: Determine the minimum depth value between the three-dimensional point cloud and the initial camera position at each viewing angle.

[0065] In a possible implementation, first, the depth value between each 3D point cloud at each viewing angle and the initial camera position at the viewing angle may be calculated respectively, and then the minimum depth value may be determined from the depth values ​​between each 3D point cloud and the initial camera position.

[0066] S202: Determine the main sight direction at each viewing angle according to the initial camera position and the camera rotation matrix at each viewing angle.

[0067] In a possible implementation, the rotation vector of each coordinate axis in the camera coordinate system at each viewing angle can be determined based on the initial camera position and the camera rotation matrix at each viewing angle, and the rotation vector of the z-axis (camera optical axis) can be determined as the main line of sight direction at each viewing angle.

[0068] S203 , the initial camera position at each viewing angle is translated by the minimum depth value along the main line of sight direction of each viewing angle to obtain the target camera position of each viewing angle.

[0069] When the minimum depth value and the main sight line direction at each viewing angle are determined, the initial camera position at each viewing angle may be translated along the main sight line direction by the minimum depth value according to the minimum depth value to obtain the target camera position at each viewing angle.

[0070] In order to better understand the translation of the camera position provided in the embodiment of the present application and the bounding box of the target object finally obtained, the difference between the bounding box of the target object provided in the present embodiment and the object bounding box in the traditional solution is explained below in conjunction with the accompanying drawings. Figure 3 The following is a comparison diagram of the bounding box of the target object provided by the embodiment of the present application and the bounding box of the object in the traditional solution. Figure 3As shown, when the initial camera position and the three-dimensional point cloud under each viewing angle are obtained by executing the above S102, the three-dimensional point clouds under multiple viewing angles together constitute the point cloud set of the target object, and the initial camera position under each viewing angle is determined respectively, that is, Figure 3 The position of the camera in the image and the minimum depth value of the 3D point cloud under the corresponding viewing angle. The minimum depth value represents the position of the point cloud set closest to the target object, that is, the position of the object surface closest to the target object. Similarly, when the initial camera position under each viewing angle is obtained, the following can be obtained based on the initial camera position under each viewing angle and the camera selection matrix: Figure 3 In this way, after determining the minimum depth value and the main sight direction at each viewing angle, the initial camera position at each viewing angle, i.e. Figure 3 The position of the camera shown in the figure is translated by the minimum depth value along the main line of sight, so that the camera at each perspective can be translated closer to the point cloud set of the target object. Therefore, the target camera position at each perspective after translation is actually the position closest to the surface of the target object. Based on the information of the bounding box determined thereby, a tighter and more accurate bounding box for the target object can be constructed.

[0071] Among them, bounding box 1 is the object bounding box in the traditional solution, and bounding box 2 is the bounding box of the target object constructed using the embodiment of the present application. Through comparison, it can be clearly found that bounding box 1 is far away from the point cloud set of the target object, and its bounding box is less compact, while bounding box 2 is closer to the point cloud set of the target object, which is more compact and fits the surface of the target object more closely.

[0072] The method provided in this embodiment can translate the initial camera position at each viewing angle along the main line of sight direction at each viewing angle based on the minimum depth value between the three-dimensional point cloud at each viewing angle and the initial camera position at the viewing angle, so as to obtain the target camera position at each viewing angle, and the target camera position at each viewing angle after the translation can be made as close to the surface of the target object as possible. Then, based on the target camera position at each viewing angle after the translation, the bounding box is estimated, which can ensure that the estimated bounding box of the target object is closer to the surface of the object, and its bounding box is more compact, thereby improving the estimation accuracy of the bounding box of the target object.

[0073] In another embodiment, the following embodiment of the present application also provides a possible implementation method for automatic estimation of a bounding box. Figure 4 A flow chart of a method for determining a bounding box of a target object in a data processing method provided in an embodiment of the present application, such as Figure 4 As shown, in S203 described in the above embodiment, the information of determining the bounding box of the target object according to the target camera positions under multiple viewing angles may include:

[0074] S401: Intersect the main sight lines under multiple viewing angles to determine the center position of the target object.

[0075] In a possible implementation, the main sight lines under multiple viewing angles may be intersected, and the center point may be estimated based on the intersection result to obtain the center position of the target object.

[0076] S402: Determine multiple distance parameters according to the target camera position and the center position under multiple viewing angles.

[0077] Among them, multiple distance parameters are used to represent the distance between the target camera position and the center position under multiple viewing angles.

[0078] S403: Determine a radius parameter of the target object according to multiple distance parameters.

[0079] The information of the bounding box includes: a center position and a radius parameter. In a possible implementation, the radius parameter of the target object may be determined according to a maximum distance parameter among a plurality of distance parameters.

[0080] In the method provided in this embodiment, the center position of the target object can be determined by intersecting the main line of sight directions under multiple perspectives, and the radius parameter of the target object can be determined based on multiple distance parameters between the target camera position and the center position under multiple perspectives, thereby realizing automatic estimation of the bounding box information of the target object, ensuring that the estimated bounding box of the target object is as close to the surface of the object as possible, making the estimated bounding box more compact.

[0081] In the above Figure 1-Figure 4 Based on the object bounding box determination method shown in any of the embodiments, the embodiments of the present application can also provide a possible implementation method for performing volume density conversion based on sampling points within the bounding box of the target object. Figure 5 A flow chart of a method for converting volume density based on a bounding box of a target object provided in an embodiment of the present application. Figure 5 As shown, after executing the above S105 to construct the bounding box of the target object in the three-dimensional space according to the information of the bounding box, the method further includes:

[0082] S501 : Sampling the three-dimensional space within the bounding box to obtain a plurality of sampling points.

[0083] In a possible implementation example, based on the bounding box of the target object obtained above, the three-dimensional space within the bounding box in the three-dimensional point cloud of the target object may be sampled to obtain multiple sampling points, and the multiple sampling points are all three-dimensional sampling points.

[0084] In a specific implementation scenario, the three-dimensional point cloud of the target object may include: the three-dimensional point cloud of the target object at multiple viewing angles. Thus, according to the bounding box, the three-dimensional space within the bounding box in the three-dimensional point cloud at each viewing angle may be sampled to obtain sampling points corresponding to each viewing angle. Then, for multiple sampling points, it may include: sampling points corresponding to multiple viewing angles. Among them, the three-dimensional point cloud at each viewing angle may be obtained based on the two-dimensional image at each viewing angle, for example. For example, for the three-dimensional point cloud at each viewing angle, according to the information of the bounding box and the preset sampling resolution, the three-dimensional space in the bounding box in the three-dimensional point cloud at each viewing angle may be sampled to obtain sampling points corresponding to each viewing angle. For example, 512 copies may be sampled for each of the xyz axes in the three-dimensional space of the bounding box, so that 512*512*512 sampling points may be sampled.

[0085] That is to say, for the multiple sampling points obtained by sampling, each sampling point has information of the corresponding viewing angle.

[0086] S502: using an initial surface distance extraction network to extract parameters of a plurality of sampling points, and obtaining parameter extraction results of the plurality of sampling points, wherein the parameter extraction result of each sampling point includes: an initial signed distance parameter of each sampling point.

[0087] For example, the initial surface distance extraction network can be a neural network with an MLP structure, which can also be called an SDF network or an implicit surface SDF neural network. The initial signed distance parameter of each sampling point can be used to represent the initial SDF value from the sampling point to the object surface.

[0088] S503. Performing volume density conversion according to the initial signed distance parameters of the multiple sampling points to obtain volume density parameters of the multiple sampling points. The volume density parameters are used to train the initial surface distance extraction network to obtain the target surface distance extraction network.

[0089] In the first implementation example, volume density conversion may be directly performed according to initial signed distance parameters of multiple sampling points to obtain volume density parameters of the multiple sampling points.

[0090] In the second implementation example, the parameter extraction result of the initial surface distance extraction network for each sampling point includes, in addition to the initial signed distance extraction parameters of the sampling point, the normal vector of the sampling point. In this example, the angle between the normal vector of each sampling point and the observation ray of each sampling point can be determined, and based on the initial signed distance parameters of the multiple sampling points, the initial signed distance parameters of the multiple sampling points and the angles corresponding to the multiple sampling points, volume density conversion is performed to obtain the volume density parameters of the multiple sampling points.

[0091] The normal vector of each sampling point can be used to represent the normal vector of each sampling point and the surface of the target object. Among them, for each sampling point, the observation ray is a ray between the camera position of each sampling point corresponding to the viewing angle to each sampling point. The camera position of each sampling point corresponding to the viewing angle can be the initial camera position at the viewing angle obtained by estimating the pose of multiple two-dimensional images.

[0092] In the specific implementation process of the second implementation example, the initial signed distance parameters of the multiple sampling points can be weighted according to the angles corresponding to the multiple sampling points, so as to achieve error correction of the signed distance parameters of each sampling point, thereby avoiding the error of the signed distance parameters of the sampling points caused by the angle of the observation ray. Then, the volume density conversion is performed according to the signed distance parameters of the multiple sampling points after the weighted processing, so as to achieve unbiased conversion of the volume density parameters of the multiple sampling points and obtain the volume density parameters of the multiple sampling points.

[0093] After obtaining the volume density parameters of the multiple sampling points, the initial surface distance extraction network can be trained based on the volume density parameters of the multiple sampling points to obtain the target surface distance extraction network.

[0094] The method provided in this embodiment can obtain multiple sampling points by sampling the three-dimensional space within the bounding box of the target object determined above, and use the initial surface distance extraction network to extract parameters of the multiple sampling points to obtain initial symbol distance parameters of the multiple sampling points, and perform volume density conversion based on the initial symbol distance parameters of the multiple sampling points to obtain volume density parameters of the multiple sampling points, which are used to train the initial surface distance extraction network to obtain the target surface distance extraction network. Since the scheme of this embodiment is based on sampling in the three-dimensional space within the bounding box of the target object obtained by automatically estimating the bounding box of the target object, the CIA in the three-dimensional space is more accurate, the sampling accuracy of the sampling points is improved, thereby improving the training accuracy of the network training based on the bounding box and the three-dimensional reconstruction accuracy of the target object, and then the completeness of the model details.

[0095] Furthermore, in a possible implementation of the present embodiment, volume density conversion can be performed according to the initial signed distance parameters of the plurality of sampling points and the angles corresponding to the plurality of sampling points to obtain volume density parameters of the plurality of sampling points. That is to say, in the process of volume density conversion, the first signed distance parameter (i.e., the initial signed distance parameter) of each sampling point can be corrected and adjusted by adding the angle corresponding to the sampling point, thereby avoiding the error of the signed distance parameter of the sampling point caused by the angle of the observation ray, and realizing unbiased conversion of the volume density parameters. The volume density conversion of the sampling points can be made as close as possible to the real surface of the object, thereby improving the training accuracy of the target surface distance extraction network, and further improving the reconstruction accuracy of the surface data of the three-dimensional model, avoiding the distortion of the model surface, and improving the completeness of the model details.

[0096] The specific implementation of volume density conversion in the data processing method provided in the above embodiment is explained below in conjunction with the accompanying drawings. Figure 6 Another method flow chart for converting volume density based on the bounding box of the target object provided in the embodiment of the present application is as follows. Figure 6 As shown above, according to the initial symbol distance parameters of the multiple sampling points and the angles corresponding to the multiple sampling points, the volume density conversion is performed to obtain the volume density parameters of the multiple sampling points, which may include:

[0097] S601: Determine a weight coefficient of each sampling point according to the cosine value of the angle.

[0098] For example, the reciprocal of the absolute value of the cosine value may be determined as the weight coefficient of each sampling point.

[0099] S602: weighting the initial signed distance parameter of each sampling point according to the weight coefficient of each sampling point to obtain a processed signed distance parameter of each sampling point.

[0100] S603: Perform volume density conversion according to the processed signed distance parameter of each sampling point to obtain a volume density parameter of each sampling point.

[0101] In the method provided in this embodiment, the volume density parameter of each sampling point can be obtained by performing volume density conversion based on the second signed distance parameter of each sampling point (i.e., the signed distance parameter of each sampling point after processing) using the probability distribution function, also known as the Logistic cumulative distribution function, or the volume density conversion formula shown below.

[0102]

[0103] Where σ(sdf) is the volume density parameter of each sampling point, β is a preset learning parameter, which is any value greater than 0; cosθ is the cosine value of the angle as described above. In this example, the SDF value is negated and divided by the absolute value of the cosine value to obtain That is, the weighted second symbol distance parameter of each sampling point, correspondingly, where, That is the weight coefficient for each sampling point.

[0104] In a specific implementation example, the weight coefficient of each sampling point can be, for example, based on the first signed distance parameters of the multiple sampling points, to determine the volume rendering weights of the multiple sampling points, and under the preset surface vertex volume rendering constraints, the parameter curve composed of the volume rendering weights of the multiple sampling points is derived, so as to obtain the expression formula of the weight coefficient of each sampling point through the cosine value of the angle. Among them, the preset surface vertex volume rendering constraints may include, for example: when the angle is maximum, the volume rendering weight corresponding to the sampling point is 0, and when the angle is minimum, the volume rendering weight corresponding to the sampling point is a maximum value. Since the expression formula of the weight coefficient of each sampling point through the cosine value of the angle is obtained under the preset surface vertex volume rendering constraints, the weight coefficient of each sampling point obtained based on the cosine value of the angle corresponding to each sampling point can realize accurate calibration and adjustment of the first signed distance parameters of each sampling point.

[0105] Then, in the method provided in this embodiment, an unbiased volume density conversion is performed based on the second signed distance parameter of each sampling point after calibration to obtain the volume density parameter of each sampling point. This can reduce the error influence on the signed distance parameter of each sampling point due to the large angle between the normal vector at the intersection of the observation ray and the surface, thereby realizing the unbiased conversion of the volume density parameters.

[0106] On the basis of the method described in any of the above embodiments, after obtaining the volume density parameters of multiple sampling points, the surface distance extraction network can be trained based on the volume density parameters of the multiple sampling points. The training process of the surface distance extraction network is illustrated below with reference to the accompanying drawings. Figure 7 A flow chart of a method for training a surface distance extraction network provided in an embodiment of the present application. Figure 7 As shown, the method also includes:

[0107] S701 . Perform volume rendering according to volume density parameters of a plurality of sampling points and rendering colors of the plurality of sampling points along corresponding observation rays to obtain rendering colors of a plurality of observation rays.

[0108] In a possible implementation, volume rendering may be performed according to the volume density parameter of each sampling point, the rendering color of each sampling point along the corresponding observation ray, and a preset sampling spacing to obtain the rendering color of an observation ray. The preset sampling spacing is the spacing between two adjacent sampling points.

[0109] S702, training an initial surface distance extraction network according to the rendering colors of the plurality of observation rays and the true value colors of the pixel positions corresponding to the plurality of observation rays in the two-dimensional image of the target object to obtain a target surface distance extraction network.

[0110] Optionally, during the process of performing the initial surface distance extraction network, the model loss function value can be calculated according to the rendering colors of multiple sample observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object, and the parameters of the initial surface distance extraction network can be adjusted according to the model loss function value and the sampling gradient descent method, and the model training can be performed according to the surface distance extraction network after the parameter adjustment until the preset stop iteration condition is met to obtain the target surface distance extraction network. The stop iteration condition can be, for example, reaching a preset maximum number of iterations, or the model loss function value reaches a preset condition.

[0111] In a possible implementation example, the rendering colors of the multiple sampling points along the corresponding observation rays involved in S701 above can be obtained in advance using a preset initial color network. That is, before performing volume rendering according to the volume density parameters of the multiple sampling points and the rendering colors of the multiple sampling points along the corresponding observation rays in S701 above to obtain the rendering colors of the multiple observation rays, the method may further include:

[0112] According to the information of the multiple sampling points and the observation rays of the multiple sampling points, a preset initial color network is used to perform color rendering on the multiple sampling points to obtain the rendering colors of the multiple sampling points along the corresponding observation rays.

[0113] For example, the initial color network may also be a neural network using an MLP structure, which may also be called a Color network. Using the initial color network, the multiple sampling points in the three-dimensional space may be rendered in color according to the input multiple sampling points and the information of the observation ray of each sampling point, and the rendering colors of the multiple sampling points along the corresponding observation ray may be obtained.

[0114] In a possible implementation example, when the rendering colors of multiple sampling points along the corresponding observation rays are obtained by color rendering using a preset initial color network, in the process of training the initial surface distance extraction network, the parameters of the initial color network can be adjusted according to the model loss function value obtained by the above calculation, so that in the process of training the initial surface distance extraction network, the initial color network can also be trained according to the model loss function value. Therefore, in the process of re-training the model according to the initial surface distance extraction network after the parameter adjustment, the model can be trained according to the initial surface distance extraction network after the parameter adjustment and the initial color network after the parameter adjustment until the preset stop iteration condition is met, and the target surface distance extraction network and the target color network can be obtained. The target color network can be used to reconstruct the target object in three dimensions using the target surface distance extraction network to obtain the three-dimensional surface data of the target object, and the rendering color of each surface vertex on the target object can also be obtained.

[0115] Optionally, when the parameters of the initial surface distance extraction network are adjusted based on the model loss function value, the preset learning parameter β in the above-mentioned volume density conversion formula can also be adjusted. In this way, when the target surface distance extraction network and the target color network are obtained through training, the target learning parameter can also be obtained. The target learning parameter can be used in the three-dimensional reconstruction process to perform unbiased volume density conversion on the target symbol distance parameter of each sampling point obtained by using the target surface distance extraction network using the volume density conversion formula with the target learning parameter to obtain the target volume density parameter of each sampling point, and then color rendering can be performed according to the target volume density parameter of each sampling point in combination with the target color network, thereby obtaining the rendering color of each surface vertex on the target object.

[0116] The method provided in this embodiment can perform volume rendering according to the volume density parameters of the multiple sampling points and the rendering colors of the multiple sampling points along the corresponding observation rays when the volume density of the multiple sampling points is obtained to obtain the rendering colors of the multiple observation rays, and train the initial surface distance extraction network according to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object to obtain the target surface distance extraction network. Since the volume density parameters of the multiple sampling points are volume density parameters obtained by performing unbiased volume density conversion based on the first signed distance parameters of the multiple sampling points and the angles corresponding to the multiple sampling points, the initial surface distance extraction network is trained based on the volume density parameters of the multiple sampling points, so that the trained target surface distance extraction network can be made more accurate, and the distance sign parameters of the extracted sampling points can be made more accurate and more precise, thereby ensuring the reconstruction accuracy of the three-dimensional model.

[0117] In the case where the object bounding box determination method provided in any of the above embodiments obtains the bounding box of the target object, the embodiment of the present application may also provide an implementation example of three-dimensional reconstruction of the target object based on the bounding box of the target object. Before executing the three-dimensional reconstruction method, the pre-trained model parameters may be imported first, and the three-dimensional reconstruction method provided by the embodiment of the present application may be executed in combination with the model. Among them, the pre-trained model parameters may at least include: the model parameters of the surface distance extraction network mentioned above. The surface distance extraction network may be the above-mentioned preset initial surface distance extraction network, or it may be a target surface distance extraction network trained by the method mentioned in any of the above embodiments, or it may be a target surface distance extraction network trained by other model training methods.

[0118] The specific implementation process of three-dimensional reconstruction is explained and illustrated in detail below with reference to the accompanying drawings. Figure 8 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application, such as Figure 8 As shown, the method may include:

[0119] S801: Obtain a bounding box of a target object.

[0120] In this embodiment, the boundary box of the target object may be, for example, a pre-defined boundary box in the above embodiment. Figure 1-Figure 4 The method provided in any embodiment estimates based on multiple two-dimensional images under multiple viewing angles to obtain a bounding box of the target object.

[0121] S802: Input multiple sampling points in the bounding box into a pre-trained surface distance extraction network to obtain target symbol distance parameters.

[0122] In an implementation example of this embodiment, the multiple sampling points within the bounding box may be obtained in a manner similar to the above-mentioned S802, or the multiple sampling points extracted based on the bounding box of the target object may be stored during the training process of the target surface distance extraction network. Here, the multiple sampling points within the bounding box obtained during the network training process may be directly obtained.

[0123] S803 . According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction on the target object to obtain three-dimensional surface data of the target object.

[0124] In a possible implementation, for example, a preset surface extraction algorithm may be used to perform isosurface extraction based on target signed distance parameters of multiple sampling points, and a three-dimensional point with a signed distance parameter of 0 may be obtained as the three-dimensional surface data of the target object. The surface extraction algorithm may be, for example, the Marching cubes algorithm described above. In a specific implementation example, a preset surface extraction algorithm may be used to perform three-dimensional reconstruction of the target object based on target signed distance parameters of multiple sampling points and a preset voxel resolution, and extract the three-dimensional surface data of the target object.

[0125] Here, the obtained three-dimensional surface data is actually a three-dimensional surface without color information, which may include: position information of each surface vertex, which can be used to render and display a colorless three-dimensional virtual model of the target object. It should be noted that after the three-dimensional surface data is obtained, the three-dimensional surface data can be stored, and according to actual business needs, the three-dimensional model data can be opened through model rendering software, so as to realize the rendering and display of the three-dimensional virtual model corresponding to the target object.

[0126] The bounding box of the target object used in the three-dimensional reconstruction method provided in the embodiment of the present application is obtained by translating the initial camera position at each perspective according to the three-dimensional point cloud at this perspective, and redetermining the bounding box of the target object determined by the target camera position at each perspective. Since the bounding box of the target object determined by this method is more compact, three-dimensional reconstruction based on the bounding box of the target object can improve the three-dimensional reconstruction accuracy of the target object and the detail completeness of the model.

[0127] In the case of obtaining the three-dimensional surface data of the target object by performing three-dimensional reconstruction using the target surface distance extraction network provided in the present embodiment, the present application embodiment may also provide a possible implementation example of color rendering of the three-dimensional surface data to determine the rendering color of each surface vertex. It should be noted that, in the case of obtaining the rendering color of each surface vertex, a colored three-dimensional virtual model of the target object can be rendered and displayed based on the three-dimensional surface data and the rendering color of each surface vertex.

[0128] In the specific application implementation process, when the target surface distance extraction network is used to obtain the target signed distance parameters of multiple sampling points, volume density conversion can be performed according to the target signed distance parameters of each sampling point and the angle between the normal vector of each sampling point and the observation ray to obtain the target volume density parameters of each sampling point; volume rendering is performed according to the target volume density parameters of the multiple sampling points and the rendering colors of the multiple sampling points along the corresponding observation rays to obtain the target rendering colors of the multiple observation rays, and then the rendering colors of each surface vertex on the target object can be obtained according to the target rendering colors of the multiple observation rays and the three-dimensional surface data.

[0129] Among them, for the rendering colors of multiple sampling points along corresponding observation rays, for example: the target color network trained in the above embodiment can be used to render the multiple sampling points in the three-dimensional space according to the information of the input multiple sampling points and the observation rays of each sampling point, so as to obtain the rendering colors of the multiple sampling points along the corresponding observation rays.

[0130] The method provided in this embodiment can realize the rendering color of each surface vertices of the target object when the three-dimensional surface data of the target object is obtained by three-dimensional reconstruction. It not only realizes the reconstruction of the three-dimensional model of the target object, but also realizes the rendering of the surface color of the target object. The final three-dimensional reconstruction data can reflect the vertex information of the object surface and the color information of each vertex on the object surface, thereby improving the model reconstruction accuracy and information completeness.

[0131] The following describes the object bounding box determination device, three-dimensional reconstruction device, equipment and storage medium provided by the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0132] Fig. 9 A schematic diagram of the structure of a device for determining an object boundary box provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the object bounding box determination device 900 may include:

[0133] The first acquisition module 901 is used to acquire multiple two-dimensional images of the target object at multiple viewing angles.

[0134] The determination module 902 is used to determine the initial camera parameters and the three-dimensional point cloud at multiple viewing angles according to the multiple two-dimensional images, wherein the initial camera parameters include: an initial camera position; determining the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle; and determining the bounding box information of the target object according to the target camera position at multiple viewing angles.

[0135] The construction module 903 is used to construct a bounding box of the target object in the three-dimensional space according to the bounding box information.

[0136] The object bounding box determination device provided in the present embodiment can determine the initial camera parameters and three-dimensional point clouds at multiple viewing angles respectively according to multiple two-dimensional images of the target object at multiple viewing angles, wherein the initial camera parameters include: an initial camera position, and then determine the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle, and then determine the information of the bounding box of the target object according to the target camera position at multiple viewing angles, and construct the bounding box of the target object in three-dimensional space according to the information of the bounding box. Since in the method provided in the embodiment of the present application, in the case of obtaining the initial camera parameters and three-dimensional point clouds at multiple viewing angles respectively based on multiple two-dimensional images, the target camera position at each viewing angle is re-determined according to the initial camera position and the three-dimensional point cloud at each viewing angle, the translation of the camera position at each viewing angle is realized, and then the object bounding box is estimated based on the target camera position at multiple viewing angles, which not only realizes the automatic estimation of the bounding box of the target object, but also makes the estimated bounding box of the target object more compact and closer to the actual object surface, thereby improving the estimation accuracy of the object bounding box, thereby improving the training accuracy of the network training based on the bounding box, and the three-dimensional reconstruction accuracy of the target object, thereby improving the detail completeness of the model.

[0137] In a feasible implementation, the initial camera parameters also include: a camera rotation matrix. The determination module 902 is specifically used to determine the minimum depth value between the three-dimensional point cloud and the initial camera position at each viewing angle; determine the main line of sight direction at each viewing angle according to the initial camera position and the camera rotation matrix at each viewing angle; translate the initial camera position at each viewing angle along the main line of sight direction of each viewing angle by the minimum depth value to obtain the target camera position at each viewing angle.

[0138] In a feasible implementation scheme, the determination module 902 is specifically used to intersect the main line of sight directions under multiple perspectives to determine the center position of the target object; determine multiple distance parameters based on the target camera position and the center position under multiple perspectives, wherein the multiple distance parameters are respectively used to represent the distance between the target camera position and the center position under multiple perspectives; determine the radius parameter of the target object from the multiple distance parameters; the bounding box information includes: the center position and the radius parameter.

[0139] In a feasible implementation manner, the determination module 902 is specifically configured to determine a radius parameter of the target object according to a maximum distance parameter among a plurality of distance parameters.

[0140] In a feasible implementation, the object bounding box determination device 900 further includes:

[0141] The sampling module is used to sample the three-dimensional space within the bounding box to obtain multiple sampling points.

[0142] A parameter extraction module is used to extract parameters of multiple sampling points using an initial surface distance extraction network to obtain initial signed distance parameters of the multiple sampling points;

[0143] The conversion module is used to perform volume density conversion according to the initial signed distance parameters of multiple sampling points to obtain volume density parameters of multiple sampling points. The volume density parameters are used to train the initial surface distance extraction network to obtain the target surface distance extraction network.

[0144] In a feasible implementation, the object bounding box determination device 900 further includes:

[0145] The training module is used to perform volume rendering according to the volume density parameters of multiple sampling points and the rendering colors of multiple sampling points along corresponding observation rays to obtain the rendering colors of multiple observation rays; according to the rendering colors of multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object, the initial surface distance extraction network is trained to obtain the target surface distance extraction network.

[0146] In a feasible implementation scheme, the training module is specifically used to calculate the model loss function value based on the rendering colors of multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object; according to the model loss function value, the initial surface distance extraction network is adjusted by sampling the gradient descent method, and the model is re-trained based on the initial surface distance extraction network after the parameter adjustment until the preset stop iteration condition is met to obtain the target surface distance extraction network.

[0147] Fig.10 A schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the 3D reconstruction device 1000 may include:

[0148] The second acquisition module 1001 is used to acquire a bounding box of a target object, wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by using the object bounding box determination method provided in any of the above embodiments.

[0149] An input module 1002 is used to input a plurality of sampling points in a bounding box into a pre-trained surface distance extraction network to obtain a target symbol distance parameter;

[0150] The reconstruction module 1003 is used to perform three-dimensional reconstruction of the target object according to the target symbol distance parameter by using a preset surface extraction algorithm to obtain three-dimensional surface data of the target object.

[0151] The bounding box of the target object used in the three-dimensional reconstruction process by the three-dimensional reconstruction device provided in the embodiment of the present application is obtained by translating the initial camera position at this perspective according to the three-dimensional point cloud at each perspective, and redetermining the bounding box of the target object determined by the target camera position at each perspective. Since the bounding box of the target object determined by this method is more compact, three-dimensional reconstruction based on the bounding box of the target object can improve the three-dimensional reconstruction accuracy of the target object and the detail completeness of the model.

[0152] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the computer device includes: a processor 1101, a storage medium 1102 and a bus 1103, the storage medium 1102 stores machine-readable instructions executable by the processor 1101, when the computer device runs a method for determining an object bounding box in the embodiment, the processor 1101 communicates with the storage medium 1102 through the bus 1103, the processor 1101 executes the machine-readable instructions, and the processor 1101 performs the preamble of the method item to perform the following steps:

[0153] Acquire multiple two-dimensional images of the target object at multiple viewing angles;

[0154] According to the multiple two-dimensional images, initial camera parameters and three-dimensional point clouds under multiple viewing angles are determined respectively, wherein the initial camera parameters include: an initial camera position;

[0155] Determine the target camera position at each viewing angle according to the initial camera position and the 3D point cloud at each viewing angle;

[0156] Determine the bounding box information of the target object according to the target camera position under multiple viewing angles;

[0157] Construct the bounding box of the target object in three-dimensional space based on the bounding box information.

[0158] In a feasible implementation, the initial camera parameters further include: a camera rotation matrix; when the processor 1101 determines the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle, it is specifically used to:

[0159] Determine the minimum depth value between the 3D point cloud and the initial camera position at each viewing angle;

[0160] Determine the main sight direction at each perspective based on the initial camera position and camera rotation matrix at each perspective;

[0161] The initial camera position at each viewing angle is translated by the minimum depth value along the main line of sight of each viewing angle to obtain the target camera position of each viewing angle.

[0162] In a feasible implementation, when the processor 1101 determines the information of the bounding box of the target object according to the target camera position under multiple viewing angles, it is specifically used to:

[0163] Intersect the main sight lines under multiple perspectives to determine the center position of the target object;

[0164] Determine a plurality of distance parameters according to the target camera position and the center position under the plurality of viewing angles, wherein the plurality of distance parameters are used to represent the distance between the target camera position and the center position under the plurality of viewing angles respectively;

[0165] The radius parameter of the target object is determined from the multiple distance parameters; the information of the bounding box includes: the center position and the radius parameter.

[0166] In a feasible implementation manner, when the processor 1101 determines the radius parameter of the target object from the multiple distance parameters, it is specifically configured to:

[0167] A radius parameter of the target object is determined according to a maximum distance parameter among the multiple distance parameters.

[0168] In a feasible implementation manner, after constructing the bounding box of the target object in the three-dimensional space according to the bounding box information, the processor 1101 is further configured to:

[0169] Sampling the three-dimensional space within the bounding box to obtain multiple sampling points;

[0170] An initial surface distance extraction network is used to extract parameters of multiple sampling points to obtain initial signed distance parameters of multiple sampling points;

[0171] According to the initial signed distance parameters of the multiple sampling points, volume density conversion is performed to obtain volume density parameters of the multiple sampling points. The volume density parameters are used to train the initial surface distance extraction network to obtain the target surface distance extraction network.

[0172] In a feasible implementation manner, after obtaining the volume density parameters of the plurality of sampling points, the processor 1101 is further configured to:

[0173] Performing volume rendering according to volume density parameters of multiple sampling points and rendering colors of multiple sampling points along corresponding observation rays to obtain rendering colors of multiple observation rays;

[0174] According to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object, the initial surface distance extraction network is trained to obtain the target surface distance extraction network.

[0175] In a feasible implementation manner, the processor 1101 is specifically used to train the initial surface distance extraction network according to the rendering colors of the plurality of observation rays and the true value colors of the pixel positions corresponding to the plurality of observation rays in the two-dimensional image of the target object to obtain the target surface distance extraction network:

[0176] Calculate the model loss function value according to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object;

[0177] According to the model loss function value, the sampling gradient descent method is used to adjust the parameters of the initial surface distance extraction network, and the model is retrained based on the initial surface distance extraction network after parameter adjustment until the preset stop iteration condition is met to obtain the target surface distance extraction network.

[0178] In the above manner, the initial camera parameters and three-dimensional point clouds at multiple viewing angles can be determined respectively according to multiple two-dimensional images of the target object at multiple viewing angles, wherein the initial camera parameters include: an initial camera position, and then the target camera position at each viewing angle is determined according to the initial camera position and the three-dimensional point cloud at each viewing angle, and then the information of the bounding box of the target object is determined according to the target camera position at multiple viewing angles, and the bounding box of the target object in three-dimensional space is constructed according to the information of the bounding box. Since in the method provided by the embodiment of the present application, in the case of obtaining the initial camera parameters and three-dimensional point clouds at multiple viewing angles respectively based on multiple two-dimensional images, the target camera position at each viewing angle is re-determined according to the initial camera position and the three-dimensional point cloud at each viewing angle, the translation of the camera position at each viewing angle is realized, and then the object bounding box is estimated based on the target camera position at multiple viewing angles, which not only realizes the automatic estimation of the bounding box of the target object, but also makes the estimated bounding box of the target object more compact and closer to the actual object surface, thereby improving the estimation accuracy of the object bounding box, thereby improving the training accuracy of the network training based on the bounding box, and the three-dimensional reconstruction accuracy of the target object, thereby improving the detail completeness of the model.

[0179] When the computer device runs the three-dimensional reconstruction method provided in the above embodiment, the processor 1101 communicates with the storage medium 1102 via the bus 1103, and the processor 1101 executes the machine-readable instructions, the preamble of the method item of the processor 1101, to perform the following steps:

[0180] Acquire a bounding box of the target object; wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by the object bounding box determination method provided in any of the above embodiments;

[0181] Input multiple sampling points in the bounding box into a pre-trained surface distance extraction network to obtain the target symbol distance parameter;

[0182] According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction of the target object to obtain the three-dimensional surface data of the target object.

[0183] Through the above method, since the bounding box of the target object used in the three-dimensional reconstruction process is determined by translating the initial camera position at this perspective according to the three-dimensional point cloud at each perspective, and re-determining the bounding box of the target object determined by the target camera position at each perspective, and since the bounding box of the target object determined by this method is more compact, three-dimensional reconstruction based on the bounding box of the target object can improve the three-dimensional reconstruction accuracy of the target object and the detail completeness of the model.

[0184] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0185] Acquire multiple two-dimensional images of the target object at multiple viewing angles;

[0186] According to the multiple two-dimensional images, initial camera parameters and three-dimensional point clouds under multiple viewing angles are determined respectively, wherein the initial camera parameters include: an initial camera position;

[0187] According to the initial camera position and the 3D point cloud at each viewing angle, the target camera position at each viewing angle is determined;

[0188] Determine the bounding box information of the target object according to the target camera position under multiple viewing angles;

[0189] Construct the bounding box of the target object in three-dimensional space based on the bounding box information.

[0190] In a feasible implementation, the initial camera parameters further include: a camera rotation matrix; when the processor determines the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle, the processor is specifically used to:

[0191] Determine the minimum depth value between the 3D point cloud and the initial camera position at each viewing angle;

[0192] Determine the main sight direction at each perspective based on the initial camera position and camera rotation matrix at each perspective;

[0193] The initial camera position at each viewing angle is translated by the minimum depth value along the main line of sight of each viewing angle to obtain the target camera position of each viewing angle.

[0194] In a feasible implementation, when the processor determines the information of the bounding box of the target object according to the target camera position under multiple viewing angles, it is specifically used to:

[0195] Intersect the main sight lines under multiple perspectives to determine the center position of the target object;

[0196] Determine a plurality of distance parameters according to the target camera position and the center position under the plurality of viewing angles, wherein the plurality of distance parameters are used to represent the distance between the target camera position and the center position under the plurality of viewing angles respectively;

[0197] The radius parameter of the target object is determined from the multiple distance parameters; the information of the bounding box includes: the center position and the radius parameter.

[0198] In a feasible implementation manner, when the processor determines the radius parameter of the target object from the multiple distance parameters, it is specifically used to:

[0199] A radius parameter of the target object is determined according to a maximum distance parameter among the multiple distance parameters.

[0200] In a feasible implementation manner, after executing the construction of the bounding box of the target object in the three-dimensional space according to the bounding box information, the processor is further configured to:

[0201] Sampling the three-dimensional space within the bounding box to obtain multiple sampling points;

[0202] An initial surface distance extraction network is used to extract parameters of multiple sampling points to obtain initial signed distance parameters of multiple sampling points;

[0203] According to the initial signed distance parameters of the multiple sampling points, volume density conversion is performed to obtain volume density parameters of the multiple sampling points. The volume density parameters are used to train the initial surface distance extraction network to obtain the target surface distance extraction network.

[0204] In a feasible implementation manner, after obtaining the volume density parameters of the plurality of sampling points, the processor is further configured to:

[0205] Performing volume rendering according to volume density parameters of multiple sampling points and rendering colors of multiple sampling points along corresponding observation rays to obtain rendering colors of multiple observation rays;

[0206] According to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object, the initial surface distance extraction network is trained to obtain the target surface distance extraction network.

[0207] In a feasible implementation, the processor, when executing training of the initial surface distance extraction network according to the rendering colors of the plurality of observation rays and the true value colors of the pixel positions corresponding to the plurality of observation rays in the two-dimensional image of the target object to obtain the target surface distance extraction network, is specifically used to:

[0208] Calculate the model loss function value according to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object;

[0209] According to the model loss function value, the sampling gradient descent method is used to adjust the parameters of the initial surface distance extraction network, and the model is retrained based on the initial surface distance extraction network after parameter adjustment until the preset stop iteration condition is met to obtain the target surface distance extraction network.

[0210] In the above manner, the initial camera parameters and three-dimensional point clouds at multiple viewing angles are determined respectively according to multiple two-dimensional images of the target object at multiple viewing angles, wherein the initial camera parameters include: an initial camera position, and then the target camera position at each viewing angle is determined according to the initial camera position and the three-dimensional point cloud at each viewing angle, and then the information of the bounding box of the target object is determined according to the target camera position at multiple viewing angles, and the bounding box of the target object in three-dimensional space is constructed according to the information of the bounding box. Since in the method provided by the embodiment of the present application, in the case of obtaining the initial camera parameters and three-dimensional point clouds at multiple viewing angles respectively based on multiple two-dimensional images, the target camera position at each viewing angle is re-determined according to the initial camera position and the three-dimensional point cloud at each viewing angle, the translation of the camera position at each viewing angle is realized, and then the object bounding box is estimated based on the target camera position at multiple viewing angles, which not only realizes the automatic estimation of the bounding box of the target object, but also makes the estimated bounding box of the target object more compact and closer to the actual object surface, thereby improving the estimation accuracy of the object bounding box, thereby improving the training accuracy of the network training based on the bounding box, and the three-dimensional reconstruction accuracy of the target object, thereby improving the detail completeness of the model.

[0211] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0212] Acquire a bounding box of the target object; wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by the object bounding box determination method provided in any of the above embodiments;

[0213] Input multiple sampling points in the bounding box into a pre-trained surface distance extraction network to obtain the target symbol distance parameter;

[0214] According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction of the target object to obtain the three-dimensional surface data of the target object.

[0215] Through the above method, since the bounding box of the target object used in the three-dimensional reconstruction process is determined by translating the initial camera position at this perspective according to the three-dimensional point cloud at each perspective, and re-determining the bounding box of the target object determined by the target camera position at each perspective, and since the bounding box of the target object determined by this method is more compact, three-dimensional reconstruction based on the bounding box of the target object can improve the three-dimensional reconstruction accuracy of the target object and the detail completeness of the model.

[0216] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0217] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0218] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0220] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0221] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0222] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for determining an object bounding box, characterized in that: include: Acquire multiple two-dimensional images of the target object at multiple viewing angles; Determine initial camera parameters and three-dimensional point clouds under the multiple viewing angles respectively according to the multiple two-dimensional images, wherein the initial camera parameters include: an initial camera position; Determine the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle; Determining information of a bounding box of the target object according to the target camera positions under the multiple viewing angles; Constructing a bounding box of the target object in three-dimensional space according to the information of the bounding box; The initial camera parameters further include: a camera rotation matrix; and determining the target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle includes: Determine the minimum depth value between the three-dimensional point cloud at each viewing angle and the initial camera position; Determine the main sight direction at each viewing angle according to the initial camera position and the camera rotation matrix at each viewing angle; The initial camera position at each viewing angle is translated by the minimum depth value along the main line of sight direction of each viewing angle to obtain the target camera position of each viewing angle.

2. The method according to claim 1, characterized in that The step of determining the bounding box information of the target object according to the target camera positions under multiple viewing angles includes: Intersecting the main sight lines under the multiple viewing angles to determine the center position of the target object; Determine a plurality of distance parameters according to the target camera positions under the plurality of viewing angles and the center position, wherein the plurality of distance parameters are respectively used to represent the distances between the target camera positions under the plurality of viewing angles and the center position; The radius parameter of the target object is determined according to the multiple distance parameters; the information of the bounding box includes: the center position and the radius parameter.

3. The method according to claim 2, characterized in that Determining the radius parameter of the target object from the multiple distance parameters includes: A radius parameter of the target object is determined according to a maximum distance parameter among the multiple distance parameters.

4. The method according to claim 1, characterized in that: After constructing the bounding box of the target object in the three-dimensional space according to the information of the bounding box, the method further includes: Sampling the three-dimensional space within the bounding box to obtain a plurality of sampling points; Using an initial surface distance extraction network, extracting parameters of a plurality of sampling points to obtain initial signed distance parameters of the plurality of sampling points; According to the initial signed distance parameters of the multiple sampling points, volume density conversion is performed to obtain the volume density parameters of the multiple sampling points, and the volume density parameters are used to train the initial surface distance extraction network to obtain the target surface distance extraction network.

5. The method according to claim 4, characterized in that After obtaining the volume density parameters of the plurality of sampling points, the method further includes: Performing volume rendering according to the volume density parameters of the plurality of sampling points and the rendering colors of the plurality of sampling points along corresponding observation rays to obtain rendering colors of the plurality of observation rays; The initial surface distance extraction network is trained according to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object to obtain the target surface distance extraction network.

6. The method according to claim 5, characterized in that The initial surface distance extraction network is trained according to the rendering colors of the multiple observation rays and the true value colors of the pixel positions corresponding to the multiple observation rays in the two-dimensional image of the target object to obtain the target surface distance extraction network, including: Calculating a model loss function value according to the rendering colors of the plurality of observation rays and the true value colors of the pixel positions corresponding to the plurality of observation rays in the two-dimensional image of the target object; According to the model loss function value, the sampling gradient descent method is used to adjust the parameters of the initial surface distance extraction network, and the model is retrained according to the initial surface distance extraction network after the parameter adjustment until the preset stop iteration condition is met to obtain the target surface distance extraction network.

7. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire a bounding box of the target object; wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by the object bounding box determination method described in any one of claims 1 to 6; Inputting a plurality of sampling points within the bounding box into a pre-trained surface distance extraction network to obtain a target symbol distance parameter; According to the target symbol distance parameter, a preset surface extraction algorithm is used to perform three-dimensional reconstruction on the target object to obtain three-dimensional surface data of the target object.

8. An object bounding box determination device, characterized in that: include: A first acquisition module is used to acquire multiple two-dimensional images of the target object at multiple viewing angles; A determination module, configured to determine initial camera parameters and three-dimensional point clouds at the multiple viewing angles respectively according to the multiple two-dimensional images, wherein the initial camera parameters include: an initial camera position; determining a target camera position at each viewing angle according to the initial camera position and the three-dimensional point cloud at each viewing angle; and determining information of a bounding box of the target object according to the target camera position at the multiple viewing angles; A construction module, configured to construct a bounding box of the target object in three-dimensional space according to the information of the bounding box; Among them, the initial camera parameters also include: a camera rotation matrix, and the determination module is specifically used to determine the minimum depth value between the three-dimensional point cloud and the initial camera position at each viewing angle; according to the initial camera position at each viewing angle and the camera rotation matrix, the main line of sight direction at each viewing angle is determined; the initial camera position at each viewing angle is translated along the main line of sight direction of each viewing angle by the minimum depth value to obtain the target camera position of each viewing angle.

9. A three-dimensional reconstruction device, characterized in that: include: A second acquisition module, configured to acquire a bounding box of a target object; wherein the bounding box is a bounding box of the target object in three-dimensional space determined in advance by the object bounding box determination method described in any one of claims 1 to 6; An input module, used for inputting a plurality of sampling points in the bounding box into a pre-trained surface distance extraction network to obtain a target symbol distance parameter; The reconstruction module is used to perform three-dimensional reconstruction on the target object according to the target symbol distance parameter by using a preset surface extraction algorithm to obtain three-dimensional surface data of the target object.

10. A computer device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the object bounding box determination method according to any one of claims 1 to 6 or the steps of the three-dimensional reconstruction method according to claim 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the object bounding box determination method according to any one of claims 1 to 6 or the three-dimensional reconstruction method according to claim 7 are executed.