3D model establishing method and device

Abstract

The invention discloses a 3D model establishment method and device, and the method comprises the steps: obtaining at least two first images which are 2D images of the same target object collected from different angles; for each first image in the at least two first images, first model information of a first 3D model corresponding to the first image is determined based on the first image, and the first model information comprises color information and first visibility information of each position point; the first visibility information represents the visible probability of the corresponding position point in the first 3D model; and aligning the first 3D models corresponding to the first images in the at least two first images, and fusing the color information of the corresponding position points based on the first visibility information of the position points in the first 3D models corresponding to the first images to obtain a target 3D model of the target object. According to the scheme, the quality of the 3D model is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and is related to but not limited to a three-dimensional (3D) model building method and device. Background Art

[0002] With the continuous development of various technologies, 3D models are widely used in game development, virtual reality, film production, digital humans and other fields. The traditional 3D software manual modeling method is time-consuming and has poor accuracy. Generating 3D models with photos can greatly improve efficiency, accuracy and speed. Photo modeling can capture the details and actual appearance of objects, making the generated models more realistic, easier to operate, highly automated, low-cost, and not restricted by time and space. The threshold of photo modeling technology is low and can be widely used.

[0003] However, the 3D model generated from one picture is generated based on a large amount of training data. The parts that cannot be captured by the picture are generated by the model itself using the training data, which is inaccurate. As a result, the 3D model generated by fusing 3D models generated from multiple pictures has the following problems: the generated basic model data is biased, the quality of the model after fusion is poor, the model is deformed, and there is a large deviation from the actual model. Summary of the invention

[0004] The present application provides a 3D model establishment method and device, and for the solution of the present application, the quality of the 3D model is improved.

[0005] The technical solution of this application is implemented as follows:

[0006] The present application provides a 3D model establishment method, the method comprising:

[0007] Obtaining at least two first images, wherein the at least two images are 2D images of the same target object acquired from different angles;

[0008] For each first image of the at least two first images, determining first model information of a first 3D model corresponding to the first image based on the first image, the first model information including color information and first visibility information of each position point, the first visibility information representing a probability that the corresponding position point is visible in the first 3D model;

[0009] The first 3D models corresponding to each first image in the at least two first images are aligned, and the color information of the corresponding position points in the first 3D models corresponding to each first image is fused based on the first visibility information of the corresponding position points to obtain the target 3D model of the target object.

[0010] The present application provides a 3D model building device, which is deployed in an electronic device and includes:

[0011] An obtaining unit, configured to obtain at least two first images, wherein the at least two images are 2D images of the same target object acquired from different angles;

[0012] a determining unit, configured to determine, for each first image of the at least two first images, first model information of a first 3D model corresponding to the first image based on the first image, the first model information comprising color information and first visibility information of each position point, the first visibility information representing a probability that the corresponding position point is visible in the first 3D model;

[0013] A fusion unit is used to align the first 3D models corresponding to each first image in the at least two first images, and fuse the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image, so as to obtain the target 3D model of the target object.

[0014] The present application also provides an electronic device, including: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned 3D model establishment method when executing the program.

[0015] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned 3D model building method is implemented.

[0016] The 3D model establishment method, apparatus, device and storage medium provided in the present application include: obtaining at least two first images, wherein the at least two images are 2D images of the same target object acquired from different angles; for each first image in the at least two first images, determining first model information of a first 3D model corresponding to the first image based on the first image, wherein the first model information includes color information and first visibility information of each position point, wherein the first visibility information represents the probability that the corresponding position point is visible in the first 3D model; aligning the first 3D models corresponding to each first image in the at least two first images, and fusing the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image, to obtain a target 3D model of the target object.

[0017] It can be seen that, for the solution of the present application, 2D images collected from different angles of the same target object are converted into 3D models, and these 3D models are aligned and fused to obtain the target 3D model of the target object, thereby solving the problem that the entire content of the target object cannot be reconstructed into a 3D model when part of the target object is blocked to a certain extent. Furthermore, for each 3D model of the 2D image, the information of the dimension of visibility information is introduced, which can better represent the spatial relationship of the feature points in the model, and the color information of the aligned 3D models at multiple angles is fused based on the visibility information, which can accurately restore the color information of each feature point of the target object, build a more accurate 3D model, and improve the quality of the reconstructed 3D model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An optional flowchart of a 3D model building method provided in an embodiment of the present application;

[0019] Figure 2 A plurality of first image schematic diagrams provided in an embodiment of the present application;

[0020] Figure 3 An optional flowchart of a 3D model building method provided in an embodiment of the present application;

[0021] Figure 4 An optional structural diagram of a generative adversarial network provided in an embodiment of the present application;

[0022] Figure 5 An optional flowchart of a 3D model building method provided in an embodiment of the present application;

[0023] Figure 6 An optional schematic diagram of a 3D model provided in an embodiment of the present application;

[0024] Figure 7 An optional flowchart of a 3D model building method provided in an embodiment of the present application;

[0025] Figure 8 for Figure 7 The schematic diagram of the invisible part of the 3D model shown;

[0026] Fig. 9 An optional flowchart of a 3D model building method provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of an optional method for capturing a second image provided in an embodiment of the present application;

[0028] Fig.11An optional structural diagram of a part of the structure of converting 2D to 3D in the generator provided in an embodiment of the present application;

[0029] Fig.12 A schematic diagram of an optional structure of a discriminator provided in an embodiment of the present application;

[0030] Fig.13 An optional schematic diagram of a single-channel color generated by a generator provided in an embodiment of the present application;

[0031] Fig.14 An optional structural schematic diagram of a 3D model building device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" are used only as examples to distinguish different objects, and do not represent a specific order for the objects, nor do they have a limitation on the order of precedence. It is understandable that "first\second\third" can be interchanged with a specific order or order of precedence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] The embodiments of the present application can provide a 3D model building method, device, equipment and storage medium. In practical applications, the 3D model building method can be implemented by a 3D model building device, and each functional entity in the 3D model building device can be collaboratively implemented by hardware resources of the electronic device, such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).

[0037] In a first aspect, an embodiment of the present application provides a 3D model building method, which is applied to a 3D model building device; wherein the 3D model building device can be deployed in an electronic device. Below, the 3D model building method provided by the embodiment of the present application is described with the electronic device as the execution subject.

[0038] Figure 1 A flowchart showing an alternative method for building a 3D model is shown. Figure 1 The process may include but is not limited to the following: Figure 1 S101 to S103 shown.

[0039] S101. Obtain at least two first images, where the at least two images are 2D images of the same target object acquired from different angles.

[0040] The electronic device can photograph the target object through the image sensor of the electronic device to collect 2D images of the target object at multiple different angles, or the electronic device can also receive 2D images of the target object collected at different angles sent by other network devices, thereby obtaining at least two first images of the target object.

[0041] In the embodiment of the present application, the target object may be a 3D object, and a 3D model simulating the target object can be established based on the target object.

[0042] In the embodiment of the present application, a first image is a 2D image of a target object captured from a certain angle, and therefore, the first image does not include a partial image of the target object. It is understandable that there may be overlapping content between the two first images.

[0043] In one example, if Figure 2 As shown, at least two first images collected for a person include Figure 2 The images 201 to 204 shown are 2D images obtained by photographing a person from different angles.

[0044] In a possible implementation manner, the contents displayed by the at least two first images of the target object can represent information of all position points of the target object.

[0045] In the embodiment of the present application, the image information of each pixel in the second image may include one or more of the following: color, depth, brightness and other information.

[0046] S102. For each first image in the at least two first images, determine first model information of a first 3D model corresponding to the first image based on the first image, the first model information including color information and first visibility information of each position point, the first visibility information representing a probability that the corresponding position point is visible in the first 3D model.

[0047] For each first image of at least two first images of the target object, the electronic device determines the first model information of the first 3D model corresponding to each first image. Among them, for a first image, the first 3D model corresponding to the first image can be understood as a 3D model that can be established based on the first image. In the embodiment of the present application, it is not necessary to construct the first 3D model corresponding to the first image, and the first model information in the first 3D model can be directly determined. The first model information can characterize the first 3D model.

[0048] The first model information includes color information and first visibility information of each position point of the first 3D model. For a position point, the color information of the position point can determine the color of the position point in the first 3D model. The first visibility information of the position point is a floating point number between 0 and 1, or can be 0 or 1, representing the probability of the position point being visible in the corresponding second image or the first 3D model. The probability of visibility can be understood as the probability or visibility of being visible to the user or the situation of being blocked, wherein the larger the value of the first visibility, the greater the probability that the position point is visible or displayed in the first 3D model, and the smaller the value of the first visibility, the smaller the probability that the position point is visible or displayed in the first 3D model.

[0049] In the embodiment of the present application, the first 3D model is composed of a plurality of model points, and each model point has a different position in the first 3D model. Here, each model point is referred to as a position point. The first model information of the first 3D model includes the position information of each model point constituting the first 3D model, thereby determining the position points constituting the first 3D model, and the first model information may also include the color information of each position point and the first visibility information, thereby determining the color of each position point in the first 3D model based on the color information of each position point.

[0050] In one example, for a position point in the first 3D model, its position information is (x, y, z), where x, y, and z are the positions of the position point on the x-axis, y-axis, and z-axis of the three dimensions of the 3D model, respectively; for the position point, the color information can be expressed as (r, g, b), representing the values ​​on the red (R) channel, green (G) channel, and blue (B) channel; for the position point, the first visibility information can be expressed as v, where v is a floating point number from 0 to 1. Therefore, for a position point in the first 3D model, it can be expressed as (x, y, z, (r, g, b, v)), the color information representing the position point (x, y, z,) is (r, g, b,), and the first visibility information is v.

[0051] In the embodiment of the present application, the color information of a position point may also be represented by other methods, and no limitation is imposed on the method of representing the color.

[0052] In a possible implementation manner, the color information of the first location point is also used to reflect the depth of the location point.

[0053] In the embodiment of the present application, different first images are 2D images collected from different angles of the same target object, and the first 3D models corresponding to the different first images represent 3D models constructed from different angles of the same target object. For a point of the target object, that is, an object point or a feature point, the position in different first 3D models may be different. Therefore, the same feature point may correspond to different position points in different first 3D models. For the feature point, the color information in different first 3D models may be the same or different, and the first visibility information in different first 3D models may be the same or different. Among them, the position information, color information and visibility information of a feature point in a first 3D model are affected by the angle at which the second image corresponding to the first 3D model is collected.

[0054] S103, aligning the first 3D models corresponding to each first image in the at least two first images, and fusing the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image, to obtain the target 3D model of the target object.

[0055] After the electronic device determines the first model information of the first 3D model corresponding to each of the at least two first images, the at least two first 3D models are aligned so that the same feature point corresponds to the same position point in different first 3D models, that is, the position points of the same feature point in different first 3D models are the same or the position information in different first 3D models is the same. It is understandable that after aligning at least two first 3D models, the position point in the first 3D model may change, that is, the position information of the model point may change. For the model point, the color information and the first visibility information of the model point do not change.

[0056] In the embodiment of the present application, feature point matching is performed on model points in different first 3D models, and a rigid body transformation matrix is ​​calculated to align the reconstruction results of different first images, namely, the first 3D models.

[0057] In an embodiment of the present application, the alignment method used to align at least two first 3D models may include a feature matching algorithm, such as: Scale Invariant Feature Transform (SIFT), SURF (Speeded-Up Robust Features), etc.

[0058] After aligning at least two first 3D models, model points with the same position information in different first 3D models, i.e., the same position points, are fused with color information of the position points in different first 3D models based on first visible information of the position points in different first 3D models to obtain a target 3D model of the target object.

[0059] In the embodiment of the present application, after the target 3D model is generated, the target 3D model may be post-processed, for example, removing noise, smoothing the model surface, and reducing model irregularities and sharp edges.

[0060] In a possible implementation, the electronic device may use a Gaussian smoothing method for post-processing. In the embodiment of the present application, there is no limitation on the method used for post-processing the target 3D model.

[0061] The 3D model building method provided in the embodiment of the present application converts 2D images collected from different angles of the same target object into 3D models, and aligns and fuses these 3D models to obtain a target 3D model of the target object, thereby solving the problem that the entire content of the target object cannot be reconstructed into a 3D model when part of the target object is blocked to a certain extent. Furthermore, for each 3D model of the 2D image, information of the dimension of visibility information is introduced, which can better represent the spatial relationship of the feature points in the model, and the color information of the aligned 3D models at multiple angles is fused based on the visibility information, which can accurately restore the color information of each feature point of the target object, build a more accurate 3D model, and improve the quality of the reconstructed 3D model.

[0062] Next, the step of fusing the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image to obtain the target 3D model of the target object in S103 is described.

[0063] In one possible implementation, reference Figure 3 As shown in the content, S103 fuses the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image, and the process of obtaining the target 3D model of the target object may include but is not limited to the following S1031 and S1032:

[0064] S1031. For the same position point, color information of the position point is fused based on first visibility information of the position point in the first 3D model corresponding to each first image to obtain target color information of the position point, wherein the position points included in the first 3D model corresponding to different first images are the same.

[0065] After at least two first 3D models are aligned, the positions included in each first 3D model are the same, and the same position point in different 3D models corresponds to the same feature point of the target object. The electronic device fuses the color information of each feature point in different first 3D models.

[0066] For a position point, the electronic device determines the color information and first visibility information of the position point in each of the at least two first 3D models based on the first model information in each of the at least two first 3D models, uses the first visibility information corresponding to each color information as the weight of the color information, and calculates the weighted average of at least two color information to obtain the target color value corresponding to the position point.

[0067] In one example, the target object corresponds to five first 3D models, namely M1 to M5. For a position point, the color information and the first visibility information of the position point in M1 are C1 and V1 respectively, the color information and the first visibility information of the position point in M2 are C2 and V2 respectively, the color information and the first visibility information of the position point in M3 are C3 and V3 respectively, the color information and the first visibility information of the position point in M4 are C4 and V4 respectively, and the color information and the first visibility information of the position point in M5 are C5 and V5 respectively. Then V1, V2, V3, V4, and V5 are used as the weights of C1, C2, C3, C4, and C5 respectively to obtain the final target color C, that is, C can be expressed as Formula 1:

[0068] C=(C1*V1+C2*V2+C3*V3+C4*V4+C5*V5) / 5 Equation 1.

[0069] S1032: Construct the target 3D model based on the target color information of each position point in the first 3D model.

[0070] After the target color information of each position point in the first 3D model is determined, a 3D model is constructed based on the position information and the target color information of each position point to obtain a target 3D model.

[0071] In an embodiment of the present application, when color information of the same position point is fused, the first visibility information is used as the weight of the corresponding color information, wherein the visibility information represents the visibility of the position point, thereby avoiding the generation of an inaccurate 3D model caused by introducing the color of the uncaptured part of the target object in the first image into the fusion of multiple 3D models, thereby improving the quality of the generated 3D model.

[0072] In a possible implementation, the color information of the position point includes sub-color information of multiple color channels, and the color information fusion process of S1031, that is, fusing the color information of the position point based on the first visibility information of the position point in each first image corresponding to the first 3D model, to obtain the target color information of the position point, includes: for each color channel in the multiple color channels, taking a weighted average of the sub-color information of the color channel at the position point by the first visibility information of the position point in each first image corresponding to the first 3D model, to obtain the target sub-color information of the color channel at the position point; wherein the target sub-color information of multiple color channels at the position point constitutes the target color information of the position point.

[0073] In an embodiment of the present application, the color information of a location point can be represented based on multiple color channels of a color model, wherein the color model can be an RGB model, a YUV model, etc. In an embodiment of the present application, there is no limitation on the type of color model used to represent the color information.

[0074] Color information may include sub-color information of each color channel in multiple color channels. For a position point, the electronic device performs weighted averaging of sub-color information of each color channel of the position point in the corresponding first 3D model based on the first visibility information of the position point in different first 3D models to obtain target sub-color information of each color channel of the position point. At this time, the target color information of the position point may include the target sub-color information of each color channel.

[0075] In one example, the color information is represented by the RGB model, and the color information can be represented by (r, g, b); the target object corresponds to five first 3D models, namely M1 to M5. For a position point, the color information and the first visibility information of the position point in M1 are C1 and V1, respectively, and C1 is represented by (r1, g1, b1). The color information and the first visibility information of the position point in M2 are C2 and V2, respectively, and C2 is represented by (r2, g2, b2). The color information of the position point in M3 is C1 and V2, respectively, and C2 is represented by (r2, g2, b2). The color information and the first visibility information of the position point in M4 are C3 and V3 respectively, and C3 is expressed as (r3, g3, b3). The color information and the first visibility information of the position point in M4 are C4 and V4 respectively, and C4 is expressed as (r4, g4, b4). The color information and the first visibility information of the position point in M5 are C5 and V5 respectively, and C5 is expressed as (r5, g5, b5). The target color C can be expressed as (r0, g0, b0,), where r0 can be expressed as Formula 2, g0 can be expressed as Formula 3, and b0 can be expressed as Formula 4:

[0076] r0=(r1*V1+r2*V2+r3*V3+r4*V4+r5*V5) / 5 Equation 2;

[0077] g0=(g1*V1+g2*V2+g3*V3+g4*V4+g5*V5) / 5 Formula 3;

[0078] b0=(b1*V1+b2*V2+b3*V3+b4*V4+b5*V5) / 5 Equation 4.

[0079] In an embodiment of the present application, color information is represented as sub-color information of multiple color channels based on a color model, and the target sub-color information of each color channel is averaged based on a weight to obtain the information of the color channel in the target color information, so that the sub-color information of different color channels is independently calculated based on the first visibility information, so that the color information of the fused target 3D model can be more accurate, thereby improving the quality of the 3D model.

[0080] Next, a process of obtaining the first model information of the first 3D model is described.

[0081] In a possible implementation, S302 determines, based on the first image, first model information of each position point in a first 3D model corresponding to the first image, including:

[0082] The first image is input into a model building network to obtain first model information of each position point in the first 3D model output by the model building network.

[0083] In an embodiment of the present application, a model building network is stored in the electronic device, and the electronic device inputs each first image of at least two first images into the model building network respectively, and the model building network outputs first model information of the first 3D model corresponding to each first image.

[0084] It can be understood that, for a 3D model, the first model information output by the model building network includes the position information, color information and first visibility information of each model point, that is, the model building network outputs the color information and first visibility information of each position point.

[0085] The model building network in the embodiment of the present application may be a neural network model. The specific structure of the model building network is not limited. The model building network in the embodiment of the present application can convert a 2D image into a 3D model and output the color information and visibility information of each position point in the 3D model.

[0086] In an embodiment of the present application, before at least two first images are input into the model building network, the first images may be preprocessed, wherein the preprocessing may include: resizing, normalization, etc., so that the model building network can better process the input first images.

[0087] In an embodiment of the present application, when adjusting the size of the first image, the principles that may be followed include but are not limited to one or more of the following: the distance between the center of the first image and the center of the area occupied by the target object is less than a set distance threshold, and the area occupied by the target object is larger than a set proportion of the area of ​​the first image, wherein the set proportion may be 50%, 60%, etc., thereby ensuring the accuracy of the information output by the model building network.

[0088] In one possible implementation, the model building network is a generator in a generative adversarial network.

[0089] In the present application, reference may be made to Figure 4 , a generative adversarial network 400 includes: a generator 401 and a discriminator 402.

[0090] The first device inputs each first image of the plurality of first images to the generator 401 , respectively, and obtains first model information of the first 3D model corresponding to each first image output by the generator 401 .

[0091] The generative adversarial network in the embodiment of the present application is a three-dimensional generative adversarial network (3D-GAN), which is a deep learning model that combines the characteristics of a convolutional neural network (CNN) and a generative adversarial network (GAN) and is used to generate or process 3D data.

[0092] The input of the generator in 3D-GAN is a two-dimensional image. The generator gradually converts the 2D image into a 3D structure through a series of 2D convolutional layers and batch normalization layers and activation functions LReLU and Sigmoid. The input of the discriminator in 3D-GAN comes from the output of the generator or real data, which is processed through 3D convolutional layers.

[0093] Below, the training process of the generative adversarial network is described when the model building network is a generator in the generative adversarial network.

[0094] The training process of generating adversarial networks can be referenced Figure 5 ,include:

[0095] S501. Acquire multiple second images of a second 3D model acquired from different angles, and second model information of a third 3D model corresponding to each second image, wherein the second model information includes color information and second visibility information of each position point, and the second visibility information represents the probability that the corresponding position point is visible in the third 3D model when the corresponding second image is acquired.

[0096] The second 3D model can be understood as a 3D model reconstructed based on the color information of all feature points of the training object. 2D images of the second 3D model are collected from different angles to obtain multiple second images. For each second image, when collecting the second image, the visibility of each position point in the second 3D model is determined, that is, the visibility information of each position point is determined, that is, the second visibility information. Based on the second visibility of each position point and the color information of each position point in the third model, the second model information of the third 3D model corresponding to the corresponding second image is obtained.

[0097] In the embodiment of the present application, the type of the training object may be the same as or different from the target object. In one example, the target object includes a person, and the training object includes a car.

[0098] In an embodiment of the present application, the second model information may include color information and second visibility information of each position point in the third 3D model. Here, the meaning and representation of the color information and the second visibility information of each position point in the third 3D model can refer to the meaning and representation of the color information and the first visibility information of each position point in the first 3D model, which will not be repeated here.

[0099] When determining the position information of each position point of the third 3D model, the position information of each model point of the second 3D model can be converted based on the difference between the angle of the coordinate axis adopted by the second 3D model and the acquisition angle of the second image to obtain the position information of each model point in the third 3D model.

[0100] In the embodiment of the present application, a method of obtaining multiple second images of the second 3D model at different angles may include at least one of the following:

[0101] 1. View the second 3D model from various angles in the display interface displaying the second 3D model and take screenshots;

[0102] 2. Take photos of the training object for constructing the second 3D model from different angles;

[0103] 3. Take photos of the 3D model from multiple angles in the 3D model editor.

[0104] When determining each second image, second visibility information on whether a model point on the second 3D model is visible at an angle at which the second image is captured may be collected simultaneously, thereby determining second model information of each third 3D model.

[0105] S502. Based on the multiple second images and the second model information of each position point in the third 3D model corresponding to each second image, train an initial generator and an initial discriminator in an initial generative adversarial network to obtain the generative adversarial network.

[0106] After the electronic device collects multiple second images and the second model information of each position point in the third 3D model corresponding to each second image, the multiple second images are used as training data, and the second model information of each position point in the third 3D model corresponding to each second image is used as a label to train the generative adversarial network, and the generator in the trained generative adversarial network is used as the model to establish the network.

[0107] Next, a process of training a generative adversarial network based on the second image and the second model information of each position point in the third 3D model corresponding to the second image is described.

[0108] In one possible implementation, reference Figure 6 S502 trains an initial generator and an initial discriminator in an initial generative adversarial network based on the plurality of second images and the second model information of each position point in the third 3D model corresponding to each second image to obtain the generative adversarial network, including:

[0109] S5021. Input the plurality of second images into the initial generator to obtain third model information of a fourth 3D model corresponding to each second image output by the initial generator, wherein the third model information includes color information and third visibility information of each position point, and the third visibility information represents a predicted probability that the corresponding position point is visible in the fourth 3D model;

[0110] S5022. For each second image in the plurality of second images, input color information of each position point in the fourth 3D model corresponding to the second image into the initial discriminator to obtain a first discrimination result output by the initial discriminator;

[0111] S5023. Based on the first discrimination result, if it is determined that the initial generative adversarial network has not converged, the parameters of the initial generative adversarial network are adjusted based on the second model information of the third 3D model corresponding to each second image in the multiple second images, and the multiple second images are continued to be input into the initial generator in the initial generative adversarial network after the parameter adjustment until the initial generative adversarial network after the parameter adjustment converges.

[0112] Each second image is input to the initial generator, and the initial generator outputs the third model data of the fourth 3D model corresponding to each predicted second image. Here, the prediction result of the initial generator includes the color information and visibility information of each position point in the predicted 3D model, i.e., the fourth 3D model, i.e., the third visibility information.

[0113] The color information generated by the initial generator is input to the initial discriminator, and the initial discriminator outputs a first discrimination result, wherein the first discrimination result output by the initial discriminator is a probability value indicating whether the input data is real data or false data generated by the initial generator. The output of the initial discriminator is close to 1, indicating that the discriminator believes that the input data is real. The output of the discriminator is close to 0, indicating that the discriminator believes that the input data is false (i.e., generated by the initial generator).

[0114] In the embodiment of the present application, the function of the initial discriminator is to map the input 3D model data to a number between 0 and 1. This value, i.e., the first discrimination result, represents the probability that the data is real data.

[0115] In a possible implementation, the color information generated by the initial generator and the color information in the second model information of the third 3D model are respectively input into the initial discriminator, and the initial discriminator outputs a first discrimination result and a second discrimination result. At this time, the first discrimination result includes a discrimination result for the color information generated by the initial generator, and the second discrimination result is a discrimination result for the color information in the second model information. At this time, whether the generative adversarial network converges can be determined based on the first discrimination result and the second discrimination result.

[0116] When the discrimination result of the initial discriminator is inaccurate, it is determined that the initial generative adversarial network has not converged. At this time, the parameters of the initial generative adversarial network are adjusted, and multiple second images are input into the initial generator again to obtain a new first discrimination result until the initial generative adversarial network converges.

[0117] Here, when the initial generative adversarial network converges, it is considered that the training of the generative adversarial network is completed, and the initial generator in the trained generative adversarial network can be used to predict the first model information for the first image.

[0118] During the training process, the initial generator and initial discriminator continuously optimize their own parameters through the adversarial process. The goal of the initial generator is to generate more and more realistic three-dimensional structures to deceive the initial discriminator, while the goal of the initial discriminator is to distinguish real data from generated data more and more accurately. Through continuous adversarial and optimization, the generator in the generative adversarial network that is finally trained can generate model data of very realistic three-dimensional models.

[0119] In a possible implementation, adjusting the parameters of the initial generative adversarial network based on the second model information of the third 3D model corresponding to each second image in the plurality of second images includes:

[0120] In the case of adjusting the parameters of the initial discriminator last time, determining a first loss based on second visibility information of each position point in the third 3D model corresponding to each second image in the plurality of second images, and adjusting the parameters of the initial generator based on the first loss;

[0121] In the case of adjusting the parameters of the initial generator last time, a second loss is determined based on the color information of each position point in the third 3D model corresponding to each second image in the multiple second images, and the parameters of the initial discriminator are adjusted based on the second loss.

[0122] In an embodiment of the present application, when the parameters of the generative adversarial network are updated, the parameters of the initial generator and the initial discriminator in the generative adversarial network are updated alternately in a loop. In one example, the parameters of the initial generator may be updated first, and the parameters of the initial discriminator may be updated next time, and then the parameters of the initial generator may be updated, and so on, until the initial generative adversarial network converges. In one example, the parameters of the initial discriminator may be updated first, and the parameters of the initial generator may be updated next time, and then the parameters of the initial discriminator may be updated, and so on, until the initial generative adversarial network converges.

[0123] When updating the parameters of the initial generator, the parameters of the initial generator are adjusted based on the first loss. When updating the parameters of the initial discriminator, the parameters of the initial discriminator are adjusted based on the second loss. The first loss is related to the second visibility information of each position point in the third 3D model corresponding to each second image in the plurality of second images, and the second loss is related to the color information of each position point in the third 3D model corresponding to each second image in the plurality of second images.

[0124] In an embodiment of the present application, the output of the generator is added with a visible dimension, i.e., visibility information, and this dimensional data is added to the calculation of the generator's loss function.

[0125] In a possible implementation, determining the first loss based on the second visibility information of each position point in the third 3D model corresponding to each second image in the plurality of second images includes:

[0126] The first loss is determined based on the second visibility information of each position point in the third 3D model corresponding to each second image in the multiple second images, and the color information and third visibility information of each position point in the fourth 3D model corresponding to each second image in the multiple second images.

[0127] In an embodiment of the present application, the input of the initial generator includes at least two second images of 2D, and the output of the initial generator includes third model information of the fourth 3D model, namely, color information and third visibility of each position point. At the same time, the data for marking the third 3D model also includes: second visibility information of each position point in the third 3D model. Therefore, the first loss can be considered based on these factors.

[0128] In one example, the first loss Loss G It can be calculated based on formula 5:

[0129] Loss G =-E p[log(D(G(p)[:,:,:,:3]))]-ylog(G(p)[:,:,:,3])-(1-y)log(1-G(p)[:,:,:,3])

[0130] Formula 5;

[0131] p is the second image used for training.

[0132] G(p) is the third model information of the fourth 3D model generated by the two-dimensional image, and its dimensions are (X×Y×Z×4). The last dimension data 4 includes color data and third visibility information, wherein X is the maximum number of elements in the x-axis direction, Y is the maximum number of elements in the y-axis direction, and Z is the maximum number of elements in the z-axis direction, which represents the position points included in the fourth 3D model.

[0133] G(p)[:,:,:,:3] represents the color information of the fourth 3D model, that is, the predicted length, width and height dimensional data of the fourth 3D model. The last dimension only takes three values ​​numbered 0, 1, and 2, representing r, g, and b, respectively.

[0134] G(p)[:,:,:,3] represents the third visibility information of the fourth 3D model, that is, the length, width and height dimensional data of the predicted 3D model. The last dimension only takes a value numbered 3, and its value v represents the visibility probability of the predicted point, ranging from 0 to 1.

[0135] y is the marking data of whether a certain point of the 3D model used for training is visible, that is, the second visibility information of each position point in the third 3D model, and its value is 0 or 1.

[0136] E represents the mathematical expectation, where E p Calculate the mathematical expectation over all input images.

[0137] Minimize-E p [log(D(G(p)[:,:,:,:3]))] is equal to maximizing D(G(p)[:,:,:,:3]), that is, making the discriminator believe that the data generated by the generator is true.

[0138] -ylog(G(p)[:,:,:,3])-(1-y)log(1-G(p)[:,:,:,3]) minimizes it, which means maximizing ylog(G(p)[:,:,:,3])+(1-y)log(1-G(p)[:,:,:,3]). When the training point is visible, y is 1, then maximizing log(G(p)[:,:,:,3], that is, maximizing G(p)[:,:,:,3], which can be understood as maximizing the visibility probability of the predicted point. When the training point is not visible, y is 0, then maximizing log(1-G(p)[:,:,:,3]), that is, minimizing G(p)[:,:,:,3], which can be understood as minimizing the visibility probability of the predicted point.

[0139] The first loss for the initial discriminator provided in the embodiment of the present application is related to the second visibility information of the marking and the third visibility information predicted by the initial discriminator. The parameters of the initial generator are adjusted based on the relationship between the marked visibility information and the predicted visibility to improve the accuracy of the third visibility information predicted by the generator.

[0140] In a possible implementation, determining the second loss based on color information of each position point in the third 3D model corresponding to each second image in the plurality of second images includes:

[0141] The second loss is determined based on the first discrimination result and the second discrimination result of each second image in the plurality of second images, wherein the second discrimination result is the output of the initial discriminator when the color information of each position point in the third 3D model corresponding to the corresponding second image is input.

[0142] In an embodiment of the present application, the input of the initial discriminator may include color information of the third 3D model, i.e., actual 3D model data, and color information of the fourth 3D model, i.e., model data of the 3D model predicted by the initial generator, thereby outputting a second discrimination result and a first discrimination result. Therefore, the first loss can be considered based on these first discrimination results and second discrimination results.

[0143] In one example, the second loss Loss for the initial discriminator is D It can be determined based on Equation 6:

[0144] Loss D =-E m [log(D(m))]-E p [log(1-D(G(p)[:,:,:,:3]))] Equation 6;

[0145] m is the color information of the third 3D model, that is, the real 3D model data.

[0146] D(m) is the discrimination result of the initial discriminator on the color information of the third 3D model.

[0147] G(p)[:,:,:,:3] is the color information of each position point of the fourth 3D model output by the initial generator.

[0148] D(G(p)[:,:,:,:3]) is the discrimination result of the initial discriminator on the color information of the fourth 3D model.

[0149] -E m Minimizing [log(D(m))] means that for all real 3D data, the discriminator hopes that D(m) is the maximum value 1.

[0150] p is the input two-dimensional image, and G(p) and G(p)[:,:,:,:3] have the same meanings as above.

[0151] -E p Minimizing [log(1-D(G(p)[:,:,:,:3]))] means that we want to maximize log(1-D(G(p)[:,:,:,:3])), that is, minimize D(G(p)[:,:,:,:3]), which can be understood as minimizing the fake 3D data generated by all input images.

[0152] In an embodiment of the present application, based on the discriminator's judgment result on the actual color information of the third 3D model and the predicted color information of the fourth 3D model, the parameters of the initial discriminator are adjusted, and the parameters of the initial discriminator can be adjusted as quickly as possible so that the initial discriminator can accurately determine whether the input data is actual color information or predicted color information.

[0153] The following describes the 3D model building method provided in the embodiment of the present application.

[0154] In the related technology, the solutions for generating 3D models from 2D models include: optical flow, point cloud processing, surface reconstruction, digital camera matrix, laser scanning, neural network and other solutions.

[0155] For optical flow, there are the following technical defects: sensitivity to motion. If the motion is complex or blurred, the optical flow calculation will become unstable or unreliable; occlusion and viewpoint changes will make the calculation difficult; multiple images are noisy and error accumulates; if the initial estimate is inaccurate, the process will fail.

[0156] For point cloud processing, there are the following technical defects: the quality of the algorithm is highly dependent on the original image and shooting conditions; the accuracy will be affected by the scanner or camera; occlusion and viewing angle limitations; multi-image noise and error accumulation; the algorithm requires professional software, professional knowledge and skills.

[0157] For surface reconstruction, there are the following technical defects: high dependence on image quality and resolution; occlusion and viewpoint limitations; noise and error accumulation; the algorithm relies on feature points and is not applicable to low contrast or sparse texture; implementation and optimization require professional knowledge and skills, and require specialized training and experience; implementation and optimization require specific software and professional knowledge.

[0158] The digital camera matrix has the following technical defects: high cost; occlusion and viewpoint limitations; implementation and optimization require professional knowledge and skills, special training and experience; implementation and optimization require specific software and professional knowledge; deployment locations are limited and mobile deployment is difficult.

[0159] For solutions such as laser scanning, there are the following technical defects: high cost; long data collection time, large or complex data collection may take several hours; implementation and optimization require professional knowledge and skills, special training and experience; processing and analysis require professional software and professional knowledge.

[0160] As for neural networks, the 3D model generated by a picture is generated based on a large amount of data training. The parts that are not captured by the picture are generated by the model itself using the training data, which is inaccurate. For example, Figure 7 The 6-sided Rubik's Cube shown above can only take a picture of 3 sides at most. Figure 8 The model images of the other three faces shown are completely based on imagination and do not correspond to the timing. In other words, the models generated by the 3D generation model for uncommon objects deviate from reality. The 3D model generated by multiple images is a 3D model generated by fusing multiple single images. The generated basic model data is biased, and the quality of the fused model is poor, the model is deformed, and the deviation from reality is large.

[0161] The 3D model building method provided in the embodiment of the present application is as follows: Fig. 9 As shown, the following steps are included:

[0162] S901, training data collection: for a determined sample 3D model, ie, a second 3D model, photograph the second 3D model from different angles to obtain sample pictures at each angle, ie, a second image, and visibility values ​​of each point in the second 3D model.

[0163] S902, training data preprocessing: preprocess each input sample image, including resizing, normalization, etc., so that the model can process it better.

[0164] S903: Training a model based on the training data, and training a GAN network through the visibility values ​​of each point in the preprocessed sample image and the 3D model corresponding to the sample image, that is, the third 3D model.

[0165] S904, generating a 3D model based on the model, and aligning and fusing the 3D model: for multiple 2D images of the same object, namely, the 2D images to be fused or the first image, they are respectively input into the generator of the GAN to obtain 3D models corresponding to the 2D images to be fused, namely, the 3D model to be fused or the first 3D model, and for each 3D model to be fused, the output result includes visibility information reflecting the visibility of each position point;

[0166] Based on the visibility information of each 3D model to be fused, multiple 3D models to be fused are fused to obtain a final 3D model.

[0167] S905: Post-processing of the generated 3D model.

[0168] Data collection for S901

[0169] In the embodiment of the present application, in the editor of the 3D model, Fig.10 As shown, the 3D model 1001 is photographed at different points and angles to obtain multiple pictures of the 3D model 1001. The 3D model editor may include blender and the like.

[0170] When the 3D model editor takes a picture of a 3D model, a 2D picture is obtained, and at the same time, information on whether points on the 3D model are visible or obscured can be obtained. This information can be automatically obtained in the editor.

[0171] Therefore, the collected data includes multiple two-dimensional images and the coordinates and color information of the 3D model points corresponding to each two-dimensional image, and information on whether the model points are visible in the currently captured image.

[0172] The coordinate points in the 3D model are represented as matrices (x, y, z), where x, y, z represent the position of the coordinate point, and the corresponding position value is a 4-tuple (r, g, b, v), where r, g, b represent colors, and v is a floating-point number representing whether the point is visible, 1 if visible, and 0 if blocked.

[0173] In the embodiment of the present application, the collected data can be expressed as (x, y, z, (r, g, b, v). The final dimension of this matrix is ​​(maximum number of elements in the x-axis direction, maximum number of elements in the y-axis direction, maximum number of elements in the z-axis direction, 4). The coordinate index is not necessarily a positive number, and the coordinate system is not necessarily the same, and data preprocessing is required later.

[0174] For the color r at (x, y, z), it can be expressed as (x coordinate index, y coordinate index, z coordinate index, 0), and its value is 0-1.

[0175] For the color g at (x, y, z), it can be expressed as (x coordinate index, y coordinate index, z coordinate index, 1), and its value is 0-1.

[0176] For the color b at (x, y, z), it can be expressed as (x coordinate index, y coordinate index, z coordinate index, 2), and its value is 0-1.

[0177] For the visibility information at (x, y, z), it can be expressed as (x coordinate index, y coordinate index, z coordinate index, 3), and its value is 0-1.

[0178] For S902 data preprocessing

[0179] Preprocess each input image, including adjusting the image size, coordinate alignment, normalization, etc., so that the model can better process it.

[0180] When preprocessing data for image models, the image size is adjusted based on one or more of the following:

[0181] Based on 1. Model input requirements: Different models may require the input image size to be a specific size. For example, some classic convolutional neural network architectures, such as AlexNet, require the input size to be 227x227, and VGGNet requires it to be 224x224.

[0182] Basis 2: Memory and computing resources: Larger image sizes mean more pixels, which will increase the number of model parameters and require more memory and computing resources. With limited resources, you may need to resize the image to a smaller size.

[0183] Basis 3: Avoid the risk of overfitting: For small datasets, using a larger image size may increase the risk of overfitting because the model can learn the details of each pixel in the image. Reducing the image size can be used as a regularization method to help the model learn more generalized features.

[0184] Criteria 4: Maintain aspect ratio: When resizing an image, it is often necessary to maintain the original aspect ratio of the image to avoid distortion of the image content. This may involve cropping or padding the image after resizing.

[0185] Basis 5, data enhancement: In some cases, resizing can also be used as a means of data enhancement. By randomly adjusting the size of the input image, the model can learn features at different scales and improve the robustness of the model.

[0186] Basis 6: Specific application requirements: Depending on the application scenario, there may be specific requirements for the size of the image. For example, in satellite image analysis, it may be necessary to maintain a certain resolution to identify specific objects.

[0187] According to 7, the proportion of the target model area in the picture is greater than a specific proportion; the center of the target model picture and the center of the model should not deviate too far, and the area occupied by the target model in the picture should be greater than 50% of the picture area. About 40% of the area can be non-target model, which is in line with the actual situation of taking pictures, increasing interference, training the model, and increasing the stability and versatility of the model.

[0188] Coordinate alignment can be understood as uniformly adjusting the coordinates of the output 3D model points to relative coordinates relative to the center of the model. If the adjusted coordinates are negative and negative matrix indexes are not supported, the new x coordinate can be made equal to the original coordinate - the original minimum x coordinate of the model. The new y and new z can be adjusted in the same way.

[0189] Build model and train for S903

[0190] 3D-GAN (3D Generative Adversarial Network) is a deep learning model that combines the features of Convolutional Neural Network (CNN) and Generative Adversarial Network (GAN) for generating or processing three-dimensional data.

[0191] The 3D-GAN network usually consists of two parts: a generator and a discriminator.

[0192] Generator:

[0193] The generator here is a variant. Its input is a prepared two-dimensional image. It gradually converts the two-dimensional image into a three-dimensional structure through a series of 2D convolutional layers and batch normalization layers and activation functions LReLU and Sigmoid.

[0194] In one example, the generator may include Fig.11 The structure shown is used to convert two-dimensional to three-dimensional. The input image is Z. After being processed by multiple convolutional layers, a model in three-dimensional space is obtained and a three-dimensional structure is output. Fig.11 The output of is a three-dimensional structure, based on which a four-dimensional matrix including visibility information can be obtained.

[0195] The generator provided in the embodiment of the present application includes an encoder-decoder architecture, wherein the encoder extracts features from the 2D image and the decoder maps these features to the 3D space. In addition, it may include additional modules to handle factors such as perspective changes and lighting conditions.

[0196] The input of the generator of the embodiment of the present application is a two-dimensional image, and the output is a matrix of X×Y×Z×4. To generate a matrix of dimension X×Y×Z×4, the simplest method is to multiply a matrix of dimension X×Y×Z×3 by a matrix of dimension 3×4. In addition, there are other ways to change the matrix dimension. For example, to generate an upper-layer weight matrix of dimension X×Y×Z×3, the lower dimension can definitely be changed. The generated matrix (X×Y×Z×3) and the upper-layer tensor product can definitely obtain the target dimension X×Y×Z×4 data.

[0197] The simple method only adds 12 elements, which is not enough to separate a visibility category field. The weight matrix dimension should be changed where the upper layer output tensor dimension is (x, y, z, 3).

[0198] This newly added weight dimension matrix or the changed weight matrix changes the model parameters of the existing 2D to 3D generator. Because the 3D model is generated from a 2D image, the labeled data contains the images and the corresponding model points. The error loss function representing the error is calculated by combining the visible data generated by the generator and the labeled data. Based on the error back propagation, the parameters of the generator can be adjusted.

[0199] The input of the discriminator comes from the generator and the real data, which is processed through a 3D convolutional layer.

[0200] The output of the discriminator is a probability value indicating whether the input data is real data or fake data generated by the generator. The output of the discriminator is close to 1, indicating that the discriminator believes that the input data is real. The output of the discriminator is close to 0, indicating that the discriminator believes that the input data is fake (that is, generated by the generator).

[0201] During the training process, the generator and the discriminator continuously optimize their own parameters through the adversarial process. The goal of the generator is to generate more and more realistic three-dimensional structures to deceive the discriminator, while the goal of the discriminator is to distinguish real data from generated data more and more accurately. Through continuous adversarial and optimization, the generator is eventually able to generate very realistic three-dimensional structures.

[0202] In the embodiment of the present application, the loss function of the generator can be expressed as Formula 5:

[0203] Loss G =-E p [log(D(G(p)[:,:,:,:3]))]-ylog(G(p)[:,:,:,3])-(1-y)log(1-G(p)[:,:,:,3])

[0204] Formula 5;

[0205] p is a two-dimensional image used for training.

[0206] G(p) is a 3D model generated from a 2D image, and its dimensions are (X×Y×Z×4). The meaning of the last dimension data can be found in the 3D model data collection in the training data collection section, where X is the maximum number of elements in the x-axis direction, Y is the maximum number of elements in the y-axis direction, and Z is the maximum number of elements in the z-axis direction.

[0207] G(p)[:,:,:,:3] indicates that the predicted 3D model has all the length, width and height dimensions, and the last dimension only takes the values ​​0, 1 and 2, representing r, g and b respectively.

[0208] G(p)[:,:,:,3] indicates that the predicted 3D model has all the length, width and height dimensions. The last dimension only takes a value numbered 3. Its value v represents the visibility probability of the predicted point, ranging from 0 to 1.

[0209] y is the labeling data of whether a certain point of the 3D model used for training is visible, and its value is 0 or 1.

[0210] E represents the mathematical expectation, where E p Calculate the mathematical expectation over all input images.

[0211] Minimize-E p [log(D(G(p)[:,:,:,:3]))] is equal to maximizing D(G(p)[:,:,:,:3]), that is, making the discriminator believe that the data generated by the generator is true.

[0212] -ylog(G(p)[:,:,:,3])-(1-y)log(1-G(p)[:,:,:,3]) minimizes it, which means maximizing ylog(G(p)[:,:,:,3])+(1-y)log(1-G(p)[:,:,:,3]). When the training point is visible, y is 1, then maximizing log(G(p)[:,:,:,3], that is, maximizing G(p)[:,:,:,3], can be understood as maximizing the visible probability of the predicted point. When the training point is not visible, y is 0, then maximizing log(1-G(p)[:,:,:,3]), that is, minimizing G(p)[:,:,:,3], can be understood as minimizing the visible probability of the predicted point.

[0213] In the embodiment of the present application, the loss function of the discriminator can be expressed as Formula 6:

[0214] Loss D =-E m [log(D(m))]-E p [log(1-D(G(p)[:,:,:,:3]))] Equation 6;

[0215] m is the real 3D model data.

[0216] -E m Minimizing [log(D(m))] means that for all real 3D data, the discriminator hopes that D(m) is the maximum value 1.

[0217] p is the input two-dimensional image, and G(p) and G(p)[:,:,:,:3] have the same meanings as above.

[0218] -E p Minimizing [log(1-D(G(p)[:,:,:,:3]))] means that we want to maximize log(1-D(G(p)[:,:,:,:3])), that is, minimize D(G(p)[:,:,:,:3]), which can be understood as minimizing the fake 3D data generated by all input images.

[0219] In the embodiment of the present application, the function of the discriminator D is to map the input 3D model data to a number between 0 and 1, and this value represents the probability that the data is real data.

[0220] For 3D-GANs processing images, the discriminator is usually a convolutional neural network, and sigmoid is used as the activation function at the end.

[0221] Fig.12 This is an example diagram of the tensor flow of each layer in the discriminator and the input and output shapes of the tensors. For the sake of example and explanation, a single color channel is used. Fig.12 As shown, the following network layers are included in sequence: layer 1: input layer, layer 2: convolution 3D layer, layer 3: batch standard layer, layer 4: activation function layer, layer 5: convolution 3D layer, layer 6: batch standard layer, layer 7: activation function layer, layer 8: convolution 3D layer, layer 9: batch standard layer, layer 10: activation function layer, layer 11: convolution 3D layer, layer 12: batch standard layer, layer 13: activation function layer, layer 14: convolution 3D layer, layer 15: batch standard layer, layer 16: activation layer, wherein the input data sizes of layers 1 to 13 are: (64, 64, 64, 1), (64, 64, 64, 1), (32 , 32,32,64), (32,32,32,64), (32,32,32,64), (16,16,16,128), (16,16,16,128), (16,16,16,128), (8,8,8,256), (8,8,8,256), (4,4,4,512), (4,4,4,512), (1,1,1,1), (1,1,1,1), where the output of the previous layer is the same size as the input of the next layer, and the output of layer 15 is (1,1,1,1). Fig.12In , None can represent batch.

[0222] An example of a single color channel (e.g. red) generator generating color information is shown below: Fig.13 shown.

[0223] In actual applications, the length of the last dimension of the generator is 4, and the value of the last dimension is (R, G, B, V), where R, G, and B represent red, green, and blue, respectively, and are floating point numbers ranging from 0 to 1. V represents whether the point is visible when the corresponding image is taken, and is a floating point number ranging from 0 to 1.

[0224] In the embodiment of the present application, the input data of the discriminator includes the 3D data generated by the generator and the collected real 3D data, and the output scalar value represents the probability that the input 3D data is real data.

[0225] The discriminator uses LeakyReLU as the activation function. The Sigmoid layer at the end of the network is used for binary classification and predicts whether the provided 3D image is real or fake.

[0226] For 3D model generation alignment and fusion in S904

[0227] Input multiple images of the target object from different angles into the generator of the trained GAN to obtain the 3D model corresponding to each image, as well as the R, G, B, and V of each position point in the 3D model.

[0228] Use feature matching algorithms (such as SIFT, SURF, etc.) to match feature points of different 3D models, calculate the rigid body transformation matrix, and align the reconstruction results of different images. The weight of the point used for alignment is the visibility value of the corresponding point.

[0229] Calculate the average value or weighted average value of each point, that is, weighted average, and fuse the reconstruction results of different images.

[0230] The improvement of fusion is that the weight of the fused point is the visibility value v of the point generated by the network, where 1 represents visible and 0 represents invisible (a floating point number, because it cannot be completely fitted into 0 / 1, it must be a floating point number distributed between 0-1, which can represent probability or weight).

[0231] In the related art, a 3D model is generated from an original image, and the color information of some 3D points that cannot be captured by this image is also generated and involved in the alignment and fusion of multiple models generated from multiple images, which is inaccurate.

[0232] The model building method in the embodiment of the present application trains the network to fit the visibility of the generated model points of the parts that cannot be captured by the image to 0, and the visibility of the model points that can be captured to 1. It can better represent the spatial relationship of the points than the training data of the generated model. The 3D model generated by multiple images can be better integrated by weighting it by its visibility, rather than calculating the average value of inaccurate points.

[0233] For the post-processing of 3D models in S905, the process includes removing noise, smoothing the model surface, reducing model irregularities and sharp edges. Gaussian smoothing method can be used for post-processing.

[0234] On the second aspect, in order to implement the above-mentioned 3D model building method, a 3D model building device of an embodiment of the present application is provided. The structural schematic diagram of the 3D model building device is described below.

[0235] like Fig.14 As shown, it includes: an obtaining unit 1401, a determining unit 1402 and a fusing unit 1403;

[0236] An obtaining unit 1401 is configured to obtain at least two first images, where the at least two images are 2D images of the same target object acquired from different angles;

[0237] a determining unit 1402, configured to determine, for each first image of the at least two first images, first model information of a first 3D model corresponding to the first image based on the first image, the first model information including color information and first visibility information of each position point, the first visibility information representing a probability that the corresponding position point is visible in the first 3D model;

[0238] The fusion unit 1403 is used to align the first 3D models corresponding to each first image in the at least two first images, and fuse the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image to obtain the target 3D model of the target object.

[0239] In some embodiments, the fusion unit 1403 is further configured to:

[0240] For the same position point, color information of the position point is fused based on first visibility information of the position point in the first 3D model corresponding to each first image to obtain target color information of the position point, wherein the position points included in the first 3D model corresponding to different first images are the same;

[0241] The target 3D model is constructed based on the target color information of each position point in the first 3D model.

[0242] In some embodiments, the fusion unit 1403 is further configured to:

[0243] For each color channel among the multiple color channels, the first visibility information of the position point in the first 3D model corresponding to each first image is weightedly averaged with the sub-color information of the color channel at the position point included in the color information of the position point to obtain the target sub-color information of the color channel at the position point; wherein the target sub-color information of the multiple color channels at the position point constitutes the target color information of the position point.

[0244] In some embodiments, the obtaining unit 1401 is further used to input the first image into a model building network to obtain first model information of each position point in the first 3D model output by the model building network.

[0245] In some embodiments, the apparatus 1400 further includes:

[0246] an acquisition unit, configured to acquire a plurality of second images acquired from different angles of the second 3D model, and second model information of the third 3D model corresponding to each second image, wherein the second model information includes color information of each position point and second visibility information, and the second visibility information represents a probability that the corresponding position point is visible in the third 3D model when the corresponding second image is acquired;

[0247] A training unit, configured to train an initial generator and an initial discriminator in an initial generative adversarial network based on the plurality of second images and second model information of each position point in a third 3D model corresponding to each second image, so as to obtain the generative adversarial network;

[0248] Among them, the model building network is a generator in a generative adversarial network.

[0249] In some embodiments, the training unit is further configured to:

[0250] Inputting the plurality of second images into the initial generator, obtaining third model information of a fourth 3D model corresponding to each second image output by the initial generator, wherein the third model information includes color information and third visibility information of each position point, and the third visibility information represents a predicted probability that the corresponding position point is visible in the fourth 3D model;

[0251] For each second image in the plurality of second images, inputting color information of each position point in the fourth 3D model corresponding to the second image into the initial discriminator to obtain a first discrimination result output by the initial discriminator;

[0252] Based on the first discrimination result, when it is determined that the initial generative adversarial network has not converged, the parameters of the initial generative adversarial network are adjusted based on the second model information of the third 3D model corresponding to each second image in the multiple second images, and the multiple second images are continued to be input into the initial generator in the initial generative adversarial network after the parameter adjustment until the initial generative adversarial network after the parameter adjustment converges.

[0253] In some embodiments, the training unit is further configured to:

[0254] In the case of adjusting the parameters of the initial discriminator last time, determining a first loss based on second visibility information of each position point in the third 3D model corresponding to each second image in the plurality of second images, and adjusting the parameters of the initial generator based on the first loss;

[0255] In the case of adjusting the parameters of the initial generator last time, a second loss is determined based on the color information of each position point in the third 3D model corresponding to each second image in the multiple second images, and the parameters of the initial discriminator are adjusted based on the second loss.

[0256] In some embodiments, the training unit is further configured to:

[0257] The first loss is determined based on the second visibility information of each position point in the third 3D model corresponding to each second image in the multiple second images, and the color information and third visibility information of each position point in the fourth 3D model corresponding to each second image in the multiple second images.

[0258] In some embodiments, the training unit is further configured to:

[0259] The second loss is determined based on the first discrimination result and the second discrimination result of each second image in the plurality of second images, wherein the second discrimination result is the output of the initial discriminator when the color information of each position point in the third 3D model corresponding to the corresponding second image is input.

[0260] It should be noted that the 3D model building device provided in the embodiment of the present application includes the various units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, MicroProcessor Unit), a digital signal processor (DSP, Digital Signal Processor) or a field programmable gate array (FPGA, Field-Programmable Gate Array), etc.

[0261] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0262] It should be noted that in the embodiment of the present application, if the above-mentioned 3D modeling method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0263] In the third aspect, in order to implement the above-mentioned 3D model building method, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the 3D model building method provided in the above-mentioned embodiment are implemented.

[0264] In a fourth aspect, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the 3D model building method provided in the above embodiment are implemented.

[0265] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0266] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0267] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0268] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0269] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0270] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0271] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0272] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0273] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A 3D model building method, the method comprising: Obtaining at least two first images, wherein the at least two images are 2D images of the same target object acquired from different angles; For each first image of the at least two first images, determining first model information of a first 3D model corresponding to the first image based on the first image, the first model information including color information and first visibility information of each position point, the first visibility information representing a probability that the corresponding position point is visible in the first 3D model; The first 3D models corresponding to each first image in the at least two first images are aligned, and the color information of the corresponding position points in the first 3D models corresponding to each first image is fused based on the first visibility information of the corresponding position points to obtain the target 3D model of the target object.

2. The method according to claim 1, wherein the step of fusing the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image to obtain the target 3D model of the target object comprises: For the same position point, color information of the position point is fused based on first visibility information of the position point in the first 3D model corresponding to each first image to obtain target color information of the position point, wherein the position points included in the first 3D model corresponding to different first images are the same; The target 3D model is constructed based on the target color information of each position point in the first 3D model.

3. According to the method of claim 1, the color information of the position point includes sub-color information of multiple color channels; the color information of the position point is fused based on the first visibility information of the position point in the first 3D model corresponding to each first image to obtain the target color information of the position point, comprising: For each color channel among the multiple color channels, the first visibility information of the position point in each first image corresponding to the first 3D model is weightedly averaged with the sub-color information of the color channel at the position point to obtain the target sub-color information of the color channel at the position point; wherein the target sub-color information of the multiple color channels at the position point constitutes the target color information of the position point.

4. The method according to any one of claims 1 to 3, wherein determining first model information of each position point in a first 3D model corresponding to the first image based on the first image comprises: The first image is input into a model building network to obtain first model information of each position point in the first 3D model output by the model building network.

5. The method according to claim 4, wherein the model building network is a generator in a generative adversarial network, and the method further comprises: Acquire a plurality of second images acquired from different angles of the second 3D model, and second model information of the third 3D model corresponding to each second image, wherein the second model information includes color information of each position point and second visibility information, and the second visibility information represents a probability that the corresponding position point is visible in the third 3D model when the corresponding second image is acquired; Based on the multiple second images and the second model information of each position point in the third 3D model corresponding to each second image, an initial generator and an initial discriminator in the initial generative adversarial network are trained to obtain the generative adversarial network.

6. The method according to claim 4, wherein: Based on the plurality of second images and the second model information of each position point in the third 3D model corresponding to each second image, an initial generator and an initial discriminator in the initial generative adversarial network are trained to obtain the generative adversarial network, including: Inputting the plurality of second images into the initial generator, obtaining third model information of a fourth 3D model corresponding to each second image output by the initial generator, wherein the third model information includes color information and third visibility information of each position point, and the third visibility information represents a predicted probability that the corresponding position point is visible in the fourth 3D model; For each second image in the plurality of second images, inputting color information of each position point in the fourth 3D model corresponding to the second image into the initial discriminator to obtain a first discrimination result output by the initial discriminator; Based on the first discrimination result, when it is determined that the initial generative adversarial network has not converged, the parameters of the initial generative adversarial network are adjusted based on the second model information of the third 3D model corresponding to each second image in the multiple second images, and the multiple second images are continued to be input into the initial generator in the initial generative adversarial network after the parameter adjustment until the initial generative adversarial network after the parameter adjustment converges.

7. The method according to claim 6, adjusting the parameters of the initial generative adversarial network based on the second model information of the third 3D model corresponding to each second image in the plurality of second images, comprising: In the case of adjusting the parameters of the initial discriminator last time, determining a first loss based on second visibility information of each position point in the third 3D model corresponding to each second image in the plurality of second images, and adjusting the parameters of the initial generator based on the first loss; In the case of adjusting the parameters of the initial generator last time, a second loss is determined based on the color information of each position point in the third 3D model corresponding to each second image in the multiple second images, and the parameters of the initial discriminator are adjusted based on the second loss.

8. The method according to claim 7, wherein determining the first loss based on the second visibility information of each position point in the third 3D model corresponding to each second image in the plurality of second images comprises: The first loss is determined based on the second visibility information of each position point in the third 3D model corresponding to each second image in the multiple second images, and the color information and third visibility information of each position point in the fourth 3D model corresponding to each second image in the multiple second images.

9. The method according to claim 7, wherein determining the second loss based on the color information of each position point in the third 3D model corresponding to each second image in the plurality of second images comprises: The second loss is determined based on the first discrimination result and the second discrimination result of each second image in the plurality of second images, wherein the second discrimination result is the output of the initial discriminator when the color information of each position point in the third 3D model corresponding to the corresponding second image is input.

10. A 3D model building device, the device being deployed in an electronic device, the device comprising: An obtaining unit, configured to obtain at least two first images, wherein the at least two images are 2D images of the same target object acquired from different angles; a determining unit, configured to determine, for each first image of the at least two first images, first model information of a first 3D model corresponding to the first image based on the first image, the first model information comprising color information and first visibility information of each position point, the first visibility information representing a probability that the corresponding position point is visible in the first 3D model; A fusion unit is used to align the first 3D models corresponding to each first image in the at least two first images, and fuse the color information of the corresponding position points based on the first visibility information of each position point in the first 3D model corresponding to each first image, so as to obtain the target 3D model of the target object.

Images