Three-dimensional model reconstruction method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for reconstructing a three-dimensional model, the method comprising: performing multi-directional image acquisition on an object to be reconstructed to obtain a plurality of images of the object to be reconstructed; inputting each of the images of the object to be reconstructed into a trained de-reflection model for de-reflection processing to obtain a plurality of de-reflection reconstructed object images; wherein the de-reflection model performs multiple down-sampling and multiple up-sampling on the input image in sequence, and fuses the result of each up-sampling and the input image through channel splicing and transposed convolution to obtain a de-reflection image; performing de-occlusion processing on the occluded area of ​​each de-reflection reconstructed object image based on the occlusion mask map of the occluded area in each de-reflection reconstructed object image through a pre-trained de-occlusion model to obtain a plurality of processed reconstructed object images; performing three-dimensional reconstruction using each of the processed reconstructed object images to obtain a three-dimensional model of the object to be reconstructed.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for reconstructing a three-dimensional model. Background Art

[0002] Currently, three-dimensional digitization has become an important means of object protection and research. By performing three-dimensional scanning on the object and using the scanned image for three-dimensional modeling, a three-dimensional model of the object can be obtained. The three-dimensional model of the object can then be used to virtually display, protect, monitor, and study the object.

[0003] Currently, the three-dimensional reconstruction of objects mainly involves first performing a three-dimensional scan on the normally placed object, and then directly using the scanned image to build a three-dimensional model of the corresponding object. The bottom of the object is then scanned upwards, and the scanned image is directly used again to build another three-dimensional model of the corresponding object. The two three-dimensional models of the object are then merged to obtain a three-dimensional model of the complete object.

[0004] However, this method of building the model twice and then fusing it is obviously relatively inefficient. The most important thing is that for special objects such as cultural relics, they are relatively fragile and important. They cannot be moved at will and can only be placed in glass display cases. Therefore, they can only be scanned across the glass display case to build a three-dimensional model. The glass display case will reflect light, and there may be other objects or patterns, which will cause reflection and occlusion problems on the acquired images, so it will affect the integrity and accuracy of the three-dimensional model built using these images. In addition, many objects cannot be flipped over and placed, so it is impossible to collect images of the bottom, which will also make it impossible to build a complete three-dimensional model. Summary of the invention

[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a three-dimensional model reconstruction method, device, equipment and storage medium to solve the problem that the prior art is low in efficiency and cannot guarantee the integrity and accuracy of the constructed three-dimensional model of the object.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] The first aspect of the present application provides a method for reconstructing a three-dimensional model, comprising:

[0008] Perform multi-directional image acquisition on the object to be reconstructed to obtain multiple images of the object to be reconstructed;

[0009] Input each of the to-be-reconstructed object images into a pre-trained de-reflection model for de-reflection processing, and obtain multiple de-reflection reconstructed object images; wherein the de-reflection model sequentially downsamples and upsamples the input image multiple times, and fuses the result of each upsampling and the input image through channel splicing and transposed convolution to obtain a de-reflection image;

[0010] Using a pre-trained de-occlusion model, based on an occlusion mask map of the occlusion area in each de-reflection reconstructed object image, de-occlusion processing is performed on the occlusion area of ​​each de-reflection reconstructed object image to obtain a plurality of processed reconstructed object images;

[0011] The processed reconstructed object images are used to perform three-dimensional reconstruction to obtain a three-dimensional model of the object to be reconstructed.

[0012] Optionally, in the above-mentioned three-dimensional model reconstruction method, the multi-directional image acquisition of the object to be reconstructed to obtain multiple images of the object to be reconstructed includes:

[0013] At a plurality of positions distributed around the object to be reconstructed, images of the object to be reconstructed corresponding to the positions are acquired; wherein the images of the object to be reconstructed corresponding to the plurality of positions include images acquired from a bird's-eye view, images acquired from a level view, and images acquired from an upward view.

[0014] Optionally, in the above-mentioned three-dimensional model reconstruction method, the step of inputting each of the to-be-reconstructed object images into a pre-trained de-reflection model for de-reflection processing to obtain a plurality of de-reflection reconstructed object images comprises:

[0015] For each of the to-be-reconstructed object images, the to-be-reconstructed object images are input into a pre-trained de-reflection model, down-sampled sequentially through each convolution layer in the encoder of the de-reflection model, and up-sampled sequentially through multiple convolution layers in the decoder to obtain a convolution result;

[0016] The convolution result, the input image and the final transposed convolution result are channel-stitched to obtain the de-reflected reconstructed object image; wherein the final transposed convolution result is obtained by sequentially channel-stitching the convolution results of each convolution layer of the decoder with the current transposed convolution result, and transposing the convolution result.

[0017] Optionally, in the above-mentioned three-dimensional model reconstruction method, the de-reflection model training method includes:

[0018] Capturing a color image of a sample object separated by a transparent material to obtain a first sample image, and capturing an image of a sample object not separated by a transparent material to obtain a first label image;

[0019] Inputting the first sample image into the de-reflection model, and processing the first sample image by the de-reflection model to obtain a current de-reflection image;

[0020] Calculate the error between the first label image and the current de-reflection image to obtain a current output result loss value, and calculate the error between the first label image and each transposed convolution result to obtain each current transposed convolution result loss value; wherein the transposed convolution result includes the result obtained by transposing the output result of the last convolution layer of the encoder in the de-reflection model, and the result of the transposed convolution of the current layer obtained by performing channel splicing and transposing convolution on each convolution layer except the last layer in the decoder in the de-reflection model with the transposed convolution result of the previous layer;

[0021] Weighting the current output result loss value and each current transposed convolution result loss value to obtain the loss value of the current de-reflection model;

[0022] Determine whether the loss value of the current de-reflection model converges;

[0023] If it is determined that the loss value of the current de-reflection model has not converged, the parameters of the de-reflection model are adjusted, and the step of inputting the first sample image into the de-reflection model is returned to execute, and the first sample image is processed by the de-reflection model to obtain a current de-reflection image;

[0024] If it is determined that the loss value of the current de-reflection model has converged, the training of the de-reflection model is terminated.

[0025] Optionally, in the above-mentioned three-dimensional model reconstruction method, the pre-trained de-occlusion model is used to perform de-occlusion processing on the occluded area of ​​each de-reflected reconstructed object image based on the occlusion mask map of the occluded area in each de-reflected reconstructed object image to obtain multiple processed reconstructed object images, including:

[0026] For each of the de-reflection reconstructed object images, the de-reflection reconstructed object images are input into the de-occlusion model, the occlusion regions in the de-reflection reconstructed object images are detected by an occlusion region detection network in the de-occlusion model, and an occlusion mask map of the occlusion regions in the de-reflection reconstructed object images is generated;

[0027] Performing de-occlusion processing on the occlusion area indicated by the occlusion mask map on the de-reflected reconstructed object image through the de-occlusion network in the de-occlusion model to obtain the processed reconstructed object image;

[0028] The de-occlusion model is trained in advance using images with and without occlusion areas, and a mask image of the existing occlusion areas.

[0029] Optionally, in the above-mentioned three-dimensional model reconstruction method, the de-occlusion model training method includes:

[0030] Collecting an original image of the object, and setting a blocking area on the original image of the object to obtain a second sample image;

[0031] Create a mask image of the set occluded area to obtain a second label image;

[0032] Iteratively training the occlusion detection network in the de-occlusion model using the second sample image and the second label image until the error between the current mask image output by the occlusion detection network and the second label image converges;

[0033] The original image of the object is used as the third label image, and the deocclusion network in the deocclusion model is iteratively trained using the second sample image and the current mask image output by the occlusion detection network until the error between the deoccluded image output by the deocclusion network and the third label image converges.

[0034] Optionally, in the above-mentioned three-dimensional model reconstruction method, performing de-occlusion processing on the occlusion area indicated by the occlusion mask map on the de-reflected reconstructed object image through the de-occlusion network in the de-occlusion model to obtain the processed reconstructed object image includes:

[0035] Inputting the de-reflected reconstructed object image and the corresponding occlusion mask map into the de-occlusion network in the de-occlusion model, and performing convolution respectively through the multiple convolution layers in the de-occlusion network to obtain a convolution feature map;

[0036] The convolution feature map is extracted and fused sequentially through the three FFC modules in the de-occlusion network;

[0037] The output of the last FFC module is combined with the convolutional feature map, and processed through multiple convolutional layers in the de-occlusion network to obtain the processed reconstructed object image.

[0038] A second aspect of the present application provides a three-dimensional model reconstruction device, comprising:

[0039] An image acquisition unit, used for acquiring multi-directional images of the object to be reconstructed to obtain multiple images of the object to be reconstructed;

[0040] A dereflection unit, used for inputting each of the images of the object to be reconstructed into a pre-trained dereflection model for dereflection processing, so as to obtain a plurality of dereflection reconstructed object images; wherein the dereflection model sequentially downsamples and upsamples the input image multiple times, and fuses the result of each upsampling and the input image through channel splicing and transposed convolution, so as to obtain a dereflected image;

[0041] a de-occlusion unit, configured to perform de-occlusion processing on the occluded area of ​​each de-reflected reconstructed object image based on an occlusion mask map of the occluded area in each de-reflected reconstructed object image by using a pre-trained de-occlusion model, so as to obtain a plurality of processed reconstructed object images;

[0042] The reconstruction unit is used to perform three-dimensional reconstruction using each of the processed reconstructed object images to obtain a three-dimensional model of the object to be reconstructed.

[0043] Optionally, in the above-mentioned three-dimensional model reconstruction device, the image acquisition unit includes:

[0044] The image acquisition subunit is used to acquire images of the object to be reconstructed corresponding to several positions distributed around the object to be reconstructed; wherein the images of the object to be reconstructed corresponding to the several positions include images acquired from a bird's-eye view, images acquired from a level view and images acquired from an upward view.

[0045] Optionally, in the above-mentioned three-dimensional model reconstruction device, the de-reflection unit includes:

[0046] The de-reflection sub-unit is used to input the image of the object to be reconstructed into a pre-trained de-reflection model for each image of the object to be reconstructed, down-sample it in turn through the convolution layers in the encoder of the de-reflection model, and up-sample it in turn through the multi-layer convolution layers of the decoder to obtain a convolution result, and channel-join the convolution result, the input image and the final transposed convolution result to obtain the de-reflection reconstructed object image; wherein the final transposed convolution result is obtained by channel-joining the convolution results of the convolution layers of the decoder and the current transposed convolution result in turn, and transposing the convolution result.

[0047] Optionally, in the above-mentioned three-dimensional model reconstruction device, it also includes:

[0048] A first training data acquisition unit is used to acquire a color image of a sample object separated by a transparent material to obtain a first sample image, and to acquire an image of a sample object not separated by a transparent material to obtain a first label image;

[0049] An input unit, used for inputting the first sample image into the de-reflection model, and processing the first sample image by the de-reflection model to obtain a current de-reflection image;

[0050] an error calculation unit, configured to calculate an error between the first label image and the current de-reflection image to obtain a current output result loss value, and to calculate an error between the first label image and each transposed convolution result to obtain each current transposed convolution result loss value; wherein the transposed convolution result includes a result obtained by performing transposed convolution on an output result of the last convolution layer of the encoder in the de-reflection model, and a result of transposed convolution of the current layer obtained by performing channel splicing and transposed convolution on each convolution layer except the last layer in the decoder in the de-reflection model with the transposed convolution result of the previous layer;

[0051] A weighted calculation unit, used for weighting the current output result loss value and each current transposed convolution result loss value to obtain a loss value of the current de-reflection model;

[0052] A judging unit, used to judge whether the loss value of the current de-reflection model converges;

[0053] an adjusting unit, configured to adjust the parameters of the de-reflection model when it is determined that the loss value of the current de-reflection model has not converged, and return to the input unit to execute the step of inputting the first sample image into the de-reflection model, and processing the first sample image through the de-reflection model to obtain a current de-reflection image;

[0054] The ending unit is used to end the training of the de-reflection model when it is determined that the loss value of the current de-reflection model has converged.

[0055] Optionally, in the above-mentioned three-dimensional model reconstruction device, the de-occlusion unit includes:

[0056] an occlusion detection unit, for inputting the de-reflection reconstructed object image into the de-occlusion model for each de-reflection reconstructed object image, detecting the occlusion area in the de-reflection reconstructed object image through the occlusion area detection network in the de-occlusion model, and generating an occlusion mask map of the occlusion area in the de-reflection reconstructed object image;

[0057] an occlusion removal unit, configured to perform a deocclusion process on the occlusion area indicated by the occlusion mask map on the de-reflected reconstructed object image through a deocclusion network in the deocclusion model, so as to obtain the processed reconstructed object image;

[0058] The de-occlusion model is trained in advance using images with and without occlusion areas, and a mask image of the existing occlusion areas.

[0059] Optionally, in the above-mentioned three-dimensional model reconstruction device, it also includes:

[0060] A second training data acquisition unit is used to acquire an original image of the object and set a blocking area on the original image of the object to obtain a second sample image;

[0061] A label making unit, used for making a mask image of a set occluding area to obtain a second label image;

[0062] A first training unit, configured to iteratively train an occlusion detection network in the de-occlusion model using the second sample image and the second label image until an error between a current mask image output by the occlusion detection network and the second label image converges;

[0063] The second training unit is used to use the original image of the object as the third label image, and use the second sample image and the current mask image output by the occlusion detection network to iteratively train the deocclusion network in the deocclusion model until the error between the deoccluded image output by the deocclusion network and the third label image converges.

[0064] Optionally, in the above-mentioned three-dimensional model reconstruction device, the de-occlusion unit performs the de-occlusion processing on the occlusion area indicated by the occlusion mask map on the de-reflected reconstructed object image through the de-occlusion network in the de-occlusion model to obtain the processed reconstructed object image, and is used to:

[0065] Inputting the de-reflected reconstructed object image and the corresponding occlusion mask map into the de-occlusion network in the de-occlusion model, and performing convolution respectively through the multiple convolution layers in the de-occlusion network to obtain a convolution feature map;

[0066] The convolution feature map is sequentially extracted and fused through the three FFC modules in the de-occlusion network;

[0067] The output of the last FFC module is combined with the convolutional feature map, and processed through multiple convolutional layers in the de-occlusion network to obtain the processed reconstructed object image.

[0068] A third aspect of the present application provides an electronic device, including:

[0069] Memory and processor;

[0070] Wherein, the memory is used to store programs;

[0071] The processor is used to execute the program, and when the program is executed, it is specifically used to implement any of the three-dimensional model reconstruction methods described above.

[0072] A fourth aspect of the present application provides a computer storage medium for storing a computer program, which, when executed, is used to implement any of the three-dimensional model reconstruction methods described above.

[0073] The present application provides a method for reconstructing a three-dimensional model, which performs multi-directional image acquisition on the object to be reconstructed to obtain multiple images of the object to be reconstructed. Since de-lighting and de-occlusion processing will be performed later, there is no need to worry about the influence of the transparent plate on the collected image. Therefore, the bottom image of the object to be reconstructed that cannot be flipped can be collected by placing it on a transparent plate, so that all-round images can be collected for the object that cannot be flipped, and then a complete three-dimensional model can be constructed at one time, thereby improving the efficiency of three-dimensional model construction. Then, each image of the object to be reconstructed is input into a pre-trained de-reflection model for de-reflection processing to obtain multiple de-reflection reconstructed objects. Among them, the de-reflection model obtains a de-reflected image by performing multiple down-sampling and multiple up-sampling on the input image in sequence, and fuses the results of each up-sampling and the input image through channel splicing and transposed convolution, thereby effectively reducing the loss of feature information, ensuring that the quality of the image after de-reflection processing is not affected, and thus the quality of the constructed three-dimensional model. Furthermore, through the pre-trained de-occlusion model, based on the occlusion mask map of the occlusion area in each de-reflection reconstructed object image, the occlusion area in each de-reflection reconstructed object image is de-occluded to obtain multiple processed reconstructed object images, thereby removing the occlusion on the image, ensuring the integrity of the object on the image, and then ensuring the integrity and accuracy of the constructed image. Finally, the processed reconstructed object image is used for three-dimensional reconstruction to obtain a three-dimensional model of the object to be reconstructed. Since each image of the object to be reconstructed contains a multi-directional image of the object, and has been de-reflected and de-occluded, the influence of reflection and occlusion on the image is effectively avoided, and the influence on the three-dimensional model constructed using these images is also avoided, so the integrity and accuracy of the three-dimensional model can be effectively guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0075] Figure 1 A flowchart of a three-dimensional model reconstruction method provided in an embodiment of the present application;

[0076] Figure 2 A flowchart of a method for training a de-reflection model provided in an embodiment of the present application;

[0077] Figure 3 It is a schematic diagram of the existing Unet network architecture;

[0078] Figure 4 A schematic diagram of a Unet network architecture provided in an embodiment of the present application;

[0079] Figure 5 A flowchart of a method for training a de-occlusion model provided in an embodiment of the present application;

[0080] Figure 6 A flowchart of a method for performing de-occlusion processing on an image through a de-occlusion network provided in an embodiment of the present application;

[0081] Figure 7 A schematic diagram of a network architecture of a de-occlusion model provided in an embodiment of the present application;

[0082] Figure 8 A schematic diagram of the architecture of a three-dimensional model reconstruction device provided in an embodiment of the present application;

[0083] Fig. 9 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0085] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0086] The present application embodiment provides a method for reconstructing a three-dimensional model. Figure 1 As shown, the following steps are included:

[0087] S101 , performing multi-directional image acquisition on an object to be reconstructed to obtain multiple images of the object to be reconstructed.

[0088] Optionally, in an embodiment of the present application, the constructed three-dimensional model needs to ensure that the color is consistent with the object, so a color image of the object to be reconstructed can be collected and the color image can be processed.

[0089] In order to construct a complete three-dimensional model of the object to be reconstructed at one time, in the embodiment of the present application, multi-directional images of the object to be reconstructed are directly collected, and specifically, the images can be omnidirectional images, that is, the bottom image is also collected at the same time.

[0090] Since the collected images need to be de-reflected and de-occluded in the subsequent process of this solution, the quality of the constructed 3D model will not be affected by placing the collected images on a transparent plate. Therefore, the bottom image of the object to be reconstructed that cannot be flipped can be collected by placing it on a transparent plate. Of course, for the object to be reconstructed that can be flipped, it can also be placed on a transparent plate for collection. Optionally, it can be placed on a transparent glass plate for collection.

[0091] Optionally, in another embodiment of the present application, a specific implementation of step S101 includes:

[0092] At a plurality of positions distributed around the object to be reconstructed, images of the object to be reconstructed corresponding to the positions are acquired.

[0093] The images of the object to be reconstructed corresponding to the plurality of positions include images collected from a bird's-eye view, images collected from a level view, and images collected from an upward view. Specifically, the preset level view angle is a horizontal angle, that is, 0 degrees, and the preset bird's-eye view angle and the preset upward view angle can be about 45 degrees plus or minus 15 degrees.

[0094] Specifically, 360-degree shooting can be performed for each shooting angle. Usually, about 80 to 120 images are shot for each viewing angle. Optionally, each rotation of 3 to 4 degrees in the horizontal direction can be regarded as a position, and an image is shot until the rotation is 360 degrees, and the preset horizontal angle is 3 degrees or 4 degrees.

[0095] Optionally, a camera may be provided at each of the three viewing angles of horizontal, upward and downward, and the three cameras are fixed on the same bracket, and the bracket may be placed on a circular slide. The middle of the circular slide is the object to be constructed, so that the bracket may be controlled to slide on the circular slide to achieve 360-degree shooting and obtain a full-face image of the object to be reconstructed.

[0096] S102 , inputting each image of the object to be reconstructed into a pre-trained de-reflection model for de-reflection processing, to obtain a plurality of de-reflection reconstructed object images.

[0097] The de-reflection model can be trained in advance using images of the same object collected with and without transparent materials, so de-reflection processing is performed using the de-reflection model.

[0098] Optionally, the method of step S101 can be specifically adopted to directly capture an image of an object to obtain an image captured without a transparent material, that is, to obtain an image without reflection, and then cover the object with a transparent glass and capture again to obtain an image captured through a transparent material, that is, to obtain an image with reflection. Then, the image captured without a transparent material is used as a label, and the image captured through a transparent material is used as a sample to train a de-reflection model, so that the image of the object to be reconstructed can be de-reflected by the de-reflection model to obtain an image of the object to be reconstructed without reflection.

[0099] It should be noted that, since the embodiment of the present application predicts a de-reflected image of RGB three channels through the de-reflection model, rather than conventionally processing a single-channel image, the de-reflection model in the present application not only performs multiple down-sampling and multiple up-sampling on the input image in sequence, that is, performs down-sampling through multiple convolutional layers in sequence, and then performs up-sampling through multiple convolutional layers. In addition, the de-reflected image is obtained by fusing the result of each up-sampling and the input image through channel splicing and transposed convolution, that is, the sampling result, the result of each up-sampling and the input original image are fused, and finally the de-reflected image is obtained, which can effectively reduce the loss of depth information and ensure the color accuracy of the processed image.

[0100] Optionally, in another embodiment of the present application, a specific implementation of step S102 includes:

[0101] For each image of the object to be reconstructed, the image of the object to be reconstructed is input into the pre-trained de-reflection model, and is down-sampled in turn through the convolution layers in the encoder of the reflectance model, and is up-sampled in turn through the multi-layer convolution layers of the decoder to obtain the convolution result, and the convolution result, the input image and the final transposed convolution result are channel-joined to obtain the de-reflection reconstructed object image.

[0102] Among them, the final transposed convolution result is obtained by sequentially splicing the convolution results of each convolution layer of the decoder with the current transposed convolution result, and transposing the convolution result.

[0103] That is, downsampling is performed through the convolution layer in the encoder, and upsampling is performed through the convolution layer in the decoder, and the convolution result is the sampling result. In addition, the convolution results of each convolution layer of the decoder are channel-joined with the current transposed convolution result, and then the joint result is transposed convolution, so as to update the current transposed convolution result. Then, it is joined with the convolution result of the next convolution layer and then transposed convolution is performed, so as to obtain the final transposed convolution result through the joint and transposed convolution processing. Then, the final transposed convolution result is joined with the original image to obtain the de-reflected image.

[0104] Optionally, another embodiment of the present application provides a method for training a de-reflection model, such as Figure 2 As shown, the following steps are included:

[0105] S201 , collecting a color image of a sample object separated by a transparent material to obtain a first sample image, and collecting an image of a sample object not separated by a transparent material to obtain a first label image.

[0106] Optionally, the method of step S101 can be used to collect a set of images of the sample object without transparent material, and these images are used as the first label image. Then, a set of images of the sample object covered by a transparent glass cover are collected in the same way, and this set of images is used as the first sample image. The photos taken by the same camera at the same angle constitute a pair of training data.

[0107] S202: Input the first sample image into a de-reflection model, and process the first sample image using the de-reflection model to obtain a current de-reflection image.

[0108] S203, calculating the error between the first label image and the current de-reflected image to obtain the current output result loss value, and calculating the error between the first label image and each transposed convolution result to obtain each current transposed convolution result loss value.

[0109] Among them, the transposed convolution result includes the result of transposed convolution of the output result of the last convolution layer of the encoder in the de-reflection model, and the result of transposed convolution of each convolution layer except the last layer in the decoder of the de-reflection model, which is obtained by channel splicing and transposed convolution of the previous layer of the transposed convolution result of the current layer.

[0110] It should be noted that in the embodiment of the present application, the reflection model adopts the Unet network architecture. Figure 3 As shown in the figure, in the existing Unet network architecture, the convolution processing is performed sequentially through the convolution layers of the encoder in the network, that is, the convolution processing is performed sequentially through the convolution layers X1-X5, and then the convolution processing is performed sequentially through the convolution layers in the decoder in combination with the output of each layer of the corresponding encoder, that is, X6-X9 is processed sequentially to obtain the final output result. This method is prone to information loss during the decoding process. However, since the current network is used to predict single-channel images, the pixel values ​​of the processed images are only 0 and 255, so even if some information is lost in time, the accuracy of the results can be guaranteed.

[0111] However, since this solution needs to predict a RGB three-channel image, the lost information will affect the accurate information of the result. Therefore, in order to utilize the feature maps output by each layer during the decoding process, the Unet network architecture is improved in the embodiment of the present application, so as to further fuse the feature maps output by each layer by using transposed convolution and channel splicing, thereby reducing the loss of deep information.

[0112] Specific as Figure 4 As shown in the figure, after the convolution layer of the last layer of the encoder, a transposed convolution layer is added to perform transposed convolution on the output result of the convolution layer of the last layer of the encoder to obtain the transposed convolution result of this layer, that is, the output result of the convolution layer X5 is transposed convolution through the transposed convolution layer Y1 to achieve upsampling of the feature map output by X5. Subsequently, the output of the first convolution layer X6 of the decoder is channel-joined with the transposed convolution result of the previous convolution layer, that is, X5, and then transposed convolution is performed through the transposed convolution layer Y2 to obtain the transposed convolution result of X6. By analogy, the transposed convolution result of the penultimate layer X8 of the decoder can be obtained. Finally, the output result of the last layer X9 of the decoder, the transposed convolution result of the penultimate layer X8, and the input image are channel-joined to obtain the final result, thereby not only utilizing the feature images obtained by each layer of the previous decoder, but also considering the features of the input image, reducing the loss of feature information, and ensuring the accuracy of the output result.

[0113] Accordingly, in order to reduce the loss of each layer output and label of the encoder, thereby reducing the loss of the de-reflection model as much as possible and ensuring the accuracy of the output result of the de-reflection model, in the embodiment of the present application, not only the error between the first label image and the current de-reflection image is calculated, but also the error between the first label image and each transposed convolution result is calculated to obtain the loss value of each current transposed convolution result.

[0114] Optionally, the error between the two images can be expressed as Mean Squared Error (MSE), which can be specifically expressed as:

[0115]

[0116] Where W, H, and C represent the width, height, and number of channels of the image, respectively, and i identifies the pixel in the image. i It represents the value of the image output by the network at pixel i, and y i It represents the value of the label image at pixel i.

[0117] S204: Weight the current output result loss value and each current transposed convolution result loss value to obtain the loss value of the current de-reflection model.

[0118] S205: Determine whether the loss value of the current de-reflection model converges.

[0119] If it is determined that the loss value of the current de-reflection model has not converged, step S206 is performed. If it is determined that the loss value of the current de-reflection model has converged, step S207 can be performed.

[0120] S206: Adjust the parameters of the de-reflection model.

[0121] Specifically, based on the loss value of the current de-reflection model, the parameters of the reflection model can be adjusted by back propagation.

[0122] Since the de-reflection model has not converged at this time, that is, the accuracy of the model has not met the requirements, after executing step S206, the process returns to executing step S202.

[0123] S207: End the training of the de-reflection model.

[0124] S103 , performing de-occlusion processing on the occluded areas of each de-reflection reconstructed object image based on the occlusion mask map of the occluded areas in each de-reflection reconstructed object image by using a pre-trained de-occlusion model, so as to obtain a plurality of processed reconstructed object images.

[0125] It should be noted that in the embodiment of the present application, the de-occlusion model is trained in advance by using images with occlusion as samples and the same images without occlusion as labels, so that the occlusion in the image of the object to be reconstructed can be removed by inputting the image of the object to be reconstructed into the de-occlusion model. In order to accurately perform de-occlusion. Specifically, when performing the processing, it is also necessary to input the occlusion mask map of the occluded area, so as to perform de-occlusion processing on the occluded area of ​​the image of the object to be reconstructed according to the instructions of the occlusion mask map.

[0126] Optionally, the occlusion mask image can be generated separately and input into the model together with the image of the object to be reconstructed for processing, or can be generated by a deocclusion model.

[0127] Optionally, in another embodiment of the present application, a specific implementation of step S103 includes:

[0128] For each de-reflected and reconstructed object image, the de-reflected and reconstructed object image is input into the de-occlusion model respectively, the occlusion area in the de-reflected and reconstructed object image is detected by the occlusion area detection network in the de-occlusion model, and an occlusion mask map of the occlusion area in the de-reflected and reconstructed object image is generated, and then the occlusion area indicated by the occlusion mask map on the de-reflected and reconstructed object image is de-occluded by the de-occlusion network in the de-occlusion model to obtain a processed reconstructed object image.

[0129] The de-occlusion model is pre-trained using images with and without occlusion areas, as well as a mask image of the existing occlusion area.

[0130] That is to say, in order to facilitate accurate reconstruction of the occluded area, there is no need to manually draw a mask map. Therefore, in an embodiment of the present application, the occlusion area detection network is combined with the de-occlusion network to form a de-occlusion model. Specifically, the occlusion area detection network can be embedded in front of the de-occlusion network to form a de-occlusion model. Of course, the occlusion area detection network and the de-occlusion network can also be two completely independent parts. Therefore, the occlusion area in the image of the object to be reconstructed can be detected by the occlusion area detection network, and the occlusion mask map of the occlusion area can be constructed, that is, the occlusion mask map corresponding to the image of the object to be reconstructed is obtained.

[0131] Optionally, another embodiment of the present application provides a method for training a de-occlusion model, such as Figure 5 As shown, the following steps are included:

[0132] S501 : Collect an original image of an object, and set a blocking area on the original image of the object to obtain a second sample image.

[0133] The original image of the object is an image obtained by directly photographing the object, without any occlusion or other images.

[0134] Optionally, an occlusion region may be set on the object original image in a random manner, so as to obtain the object original image with occlusion, and these images are used as the second sample images.

[0135] S502: Create a mask image of the set occluding area to obtain a second label image.

[0136] S503: Using the second sample image and the second label image, iteratively train the occlusion detection network in the de-occlusion model until the error between the current mask image output by the occlusion detection network and the second label image converges.

[0137] It should be noted that in the embodiment of the present application, the occlusion detection network is trained first, and after the occlusion detection network is trained, the occlusion detection network is trained again.

[0138] Specifically, the second sample image is input into the occlusion detection network, and the occlusion area of ​​the second sample image is detected by the occlusion detection network to generate a corresponding mask image, that is, output the current mask image. Then, the error between the generated current mask image and the second label image is calculated, and it is determined whether the error converges. If the error has not converged, it is necessary to adjust the parameters of the occlusion detection network, and return to the second training again, so as to continuously perform iterative training until the error converges.

[0139] S504: Use the original image of the object as the third label image, and use the second sample image and the current mask image output by the occlusion detection network to iteratively train the deocclusion network in the deocclusion model until the error between the deoccluded image output by the deocclusion network and the third label image converges.

[0140] Specifically, the second sample image and the current mask image output by the occlusion detection network are input into the deocclusion network in the deocclusion model, and the deoccluded image is output, that is, the image after the second sample image is deoccluded is output. Then, the error between the deoccluded image and the third label image is calculated, and iterative training is performed based on the error.

[0141] Optionally, in another embodiment of the present application, a specific real-time method is used to perform de-occlusion processing on the occlusion area indicated by the occlusion mask on the de-reflected reconstructed object image through the de-occlusion network in the de-occlusion model to obtain a processed reconstructed object image, such as Figure 6 As shown, the following steps are included:

[0142] S601, inputting the de-reflected reconstructed object image and its corresponding occlusion mask map into a de-occlusion network in a de-occlusion model, and performing convolution respectively through the multiple convolution layers in the de-occlusion network to obtain a convolution feature map.

[0143] S602, extracting and fusing the convolutional feature map through the three FFC modules in the de-occlusion network in turn.

[0144] Among them, the FFC module is the fast Fourier convolution module (Fast Fourier Convolution, FFC).

[0145] It should be noted that in the embodiment of the present application, the de-occlusion network adopts the DeepFillV1 network. The continuous hole convolution layer in the DeepFillV1 network is mainly used to improve the receptive field, so that the network can consider the overall characteristics of the image to repair the mask area. In order to improve the accuracy of the output results of the network, the FFC module is introduced in the embodiment of the present application, and the FFC residual module is designed, that is, an FFC residual module composed of three FFC modules is designed on the basis of the FFC module, and the FFC residual module is used to replace the continuous hole convolution layer in the DeepFillV1 network, so as to effectively fuse local features and global features while maintaining the receptive field.

[0146] Therefore, in the present embodiment, Figure 7 As shown in the figure, the de-occlusion model processes the image of the object to be reconstructed. The occlusion area is first detected through the occlusion detection network to obtain the corresponding mask image. Then the corresponding mask image and the image of the object to be reconstructed are input into the occlusion network, and convolved through multiple convolutional layers respectively, and then processed through three FFC modules, and then the final input of the FFC module is combined with the initial output, and finally the combined result is convolved through multiple convolutional layers to obtain the de-occluded image.

[0147] S603: Process the output of the last FFC module and the convolutional feature map through multiple convolutional layers in the de-occlusion network to obtain a processed reconstructed object image.

[0148] S104 , performing three-dimensional reconstruction using each of the processed reconstructed object images to obtain a three-dimensional model of the object to be reconstructed.

[0149] Optionally, a neural radiation field series algorithm NERF, a 3D Gaussian scattering series algorithm 3DGS, or reality capture software such as Reality Capture can be used to reconstruct the images of the object to be reconstructed after de-reflection processing and de-occlusion processing to obtain a three-dimensional model of the object to be reconstructed.

[0150] The present application provides a method for reconstructing a three-dimensional model, which performs multi-directional image acquisition on the object to be reconstructed to obtain multiple images of the object to be reconstructed. Since de-lighting and de-occlusion processing will be performed later, there is no need to worry about the influence of the transparent plate on the collected image. Therefore, the bottom image of the object to be reconstructed that cannot be flipped can be collected by placing it on a transparent plate, so that all-round images can be collected for the object that cannot be flipped, and then a complete three-dimensional model can be constructed at one time, thereby improving the efficiency of three-dimensional model construction. Then, each image of the object to be reconstructed is input into a pre-trained de-reflection model for de-reflection processing to obtain multiple de-reflection reconstructed objects. Among them, the de-reflection model obtains a de-reflected image by performing multiple down-sampling and multiple up-sampling on the input image in sequence, and fuses the results of each up-sampling and the input image through channel splicing and transposed convolution, thereby effectively reducing the loss of feature information, ensuring that the quality of the image after de-reflection processing is not affected, and thus the quality of the constructed three-dimensional model. Furthermore, through the pre-trained de-occlusion model, based on the occlusion mask map of the occlusion area in each de-reflection reconstructed object image, the occlusion area in each de-reflection reconstructed object image is de-occluded to obtain multiple processed reconstructed object images, thereby removing the occlusion on the image, ensuring the integrity of the object on the image, and then ensuring the integrity and accuracy of the constructed image. Finally, the processed reconstructed object image is used for three-dimensional reconstruction to obtain a three-dimensional model of the object to be reconstructed. Since each image of the object to be reconstructed contains a multi-directional image of the object, and has been de-reflected and de-occluded, the influence of reflection and occlusion on the image is effectively avoided, and the influence on the three-dimensional model constructed using these images is also avoided, so the integrity and accuracy of the three-dimensional model can be effectively guaranteed.

[0151] Another embodiment of the present application provides a three-dimensional model reconstruction device, such as Figure 8 As shown, including:

[0152] The image acquisition unit 801 is used to acquire multi-directional images of the object to be reconstructed to obtain multiple images of the object to be reconstructed.

[0153] The de-reflection unit 802 is used to input each image of the object to be reconstructed into a pre-trained de-reflection model to perform de-reflection processing, so as to obtain multiple de-reflection reconstructed object images.

[0154] Among them, the de-reflection model obtains a de-reflected image by performing multiple down-sampling and multiple up-sampling on the input image in sequence, and fusing the result of each up-sampling and the input image through channel splicing and transposed convolution.

[0155] The de-occlusion unit 803 is used to perform de-occlusion processing on the occluded areas of each de-reflected reconstructed object image based on the occlusion mask map of the occluded areas in each de-reflected reconstructed object image through a pre-trained de-occlusion model to obtain multiple processed reconstructed object images.

[0156] The reconstruction unit 804 is used to perform three-dimensional reconstruction using each processed reconstructed object image to obtain a three-dimensional model of the object to be reconstructed.

[0157] Optionally, in the three-dimensional model reconstruction device provided in another embodiment of the present application, the image acquisition unit includes:

[0158] The image acquisition subunit is used to acquire images of the object to be reconstructed corresponding to a plurality of positions distributed around the object to be reconstructed; wherein the images of the object to be reconstructed corresponding to the plurality of positions include images acquired from a bird's-eye view, images acquired from a level view, and images acquired from an upward view.

[0159] Optionally, in the three-dimensional model reconstruction device provided in another embodiment of the present application, the de-reflection unit includes:

[0160] The de-reflection sub-unit is used for the de-reflection sub-unit, which is used for inputting the image of the object to be reconstructed into the pre-trained de-reflection model for each image of the object to be reconstructed, down-sampling through the convolution layers of the encoder of the reflectance model in sequence, and up-sampling through the multi-layer convolution layers of the decoder in sequence to obtain the convolution result, and channel-joining the convolution result, the input image and the final transposed convolution result to obtain the de-reflection reconstructed object image. The final transposed convolution result is obtained by channel-joining the convolution results of the convolution layers of the decoder and the current transposed convolution result in sequence, and transposing the convolution result.

[0161] Optionally, in another embodiment of the present application, the three-dimensional model reconstruction device further includes:

[0162] The first training data acquisition unit is used to acquire a color image of a sample object separated by a transparent material to obtain a first sample image, and to acquire an image of a sample object not separated by a transparent material to obtain a first label image.

[0163] The input unit is used to input the first sample image into the de-reflection model, and process the first sample image through the de-reflection model to obtain a current de-reflection image.

[0164] The error calculation unit is used to calculate the error between the first label image and the current de-reflection image to obtain the current output result loss value, and calculate the error between the first label image and each transposed convolution result to obtain each current transposed convolution result loss value. The transposed convolution result includes the result obtained by transposing the output result of the last convolution layer of the encoder in the de-reflection model, and the result of the transposed convolution of each convolution layer except the last layer in the decoder in the de-reflection model after channel splicing with the transposed convolution result of the previous layer and performing transposed convolution.

[0165] The weighted calculation unit is used to weight the current output result loss value with each current transposed convolution result loss value to obtain the loss value of the current de-reflection model.

[0166] The judgment unit is used to judge whether the loss value of the current de-reflection model converges.

[0167] The adjustment unit is used to adjust the parameters of the de-reflection model when it is determined that the loss value of the current de-reflection model has not converged, and return to the input unit to execute inputting the first sample image into the de-reflection model, and process the first sample image through the de-reflection model to obtain the current de-reflection image.

[0168] The ending unit is used to end the training of the de-reflection model when it is determined that the loss value of the current de-reflection model has converged.

[0169] Optionally, in the three-dimensional model reconstruction device provided in another embodiment of the present application, the de-occlusion unit includes:

[0170] The occlusion detection unit is used to input the de-reflection reconstructed object image into the de-occlusion model for each de-reflection reconstructed object image, detect the occlusion area in the de-reflection reconstructed object image through the occlusion area detection network in the de-occlusion model, and generate an occlusion mask map of the occlusion area in the de-reflection reconstructed object image.

[0171] The occlusion removal unit is used to perform a de-occlusion process on the occlusion area indicated by the occlusion mask map on the de-reflection reconstructed object image through a de-occlusion network in the de-occlusion model to obtain a processed reconstructed object image.

[0172] The de-occlusion model is pre-trained using images with and without occlusion areas, as well as a mask image of the existing occlusion area.

[0173] Optionally, in another embodiment of the present application, the three-dimensional model reconstruction device further includes:

[0174] The second training data acquisition unit is used to acquire an original image of the object and set a blocking area on the original image of the object to obtain a second sample image.

[0175] The label making unit is used to make a mask image of the set occluding area to obtain a second label image.

[0176] The first training unit is used to iteratively train the occlusion detection network in the de-occlusion model using the second sample image and the second label image until the error between the current mask image output by the occlusion detection network and the second label image converges.

[0177] The second training unit is used to use the original image of the object as the third label image, and use the second sample image and the current mask image output by the occlusion detection network to iteratively train the deocclusion network in the deocclusion model until the error between the deoccluded image output by the deocclusion network and the third label image converges.

[0178] Optionally, in the three-dimensional model reconstruction device provided in another embodiment of the present application, the de-occlusion unit performs de-occlusion processing on the occlusion area indicated by the occlusion mask map on the de-reflection reconstructed object image through the de-occlusion network in the de-occlusion model to obtain the processed reconstructed object image, which is used to:

[0179] The de-reflected reconstructed object image and its corresponding occlusion mask map are input into the de-occlusion network in the de-occlusion model, and convolution is performed respectively through the multiple convolution layers in the de-occlusion network to obtain a convolution feature map.

[0180] The convolutional feature maps are extracted and fused in turn through the three FFC modules in the de-occlusion network.

[0181] The output of the last FFC module is combined with the convolutional feature map and processed through multiple convolutional layers in the de-occlusion network to obtain the processed reconstructed object image.

[0182] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.

[0183] Another embodiment of the present application provides an electronic device, such as Fig. 9 As shown, including:

[0184] Memory 901 and processor 902 .

[0185] The memory 901 is used to store programs.

[0186] The processor 902 is used to execute the program stored in the memory 901. When the program is executed, it is specifically used to implement the three-dimensional model reconstruction method provided by any one of the above embodiments.

[0187] Another embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is used to implement the three-dimensional model reconstruction method provided in any one of the above embodiments.

[0188] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0189] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0190] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for reconstructing a three-dimensional model, characterized in that: include: Perform multi-directional image acquisition on the object to be reconstructed to obtain multiple images of the object to be reconstructed; Input each of the to-be-reconstructed object images into a pre-trained de-reflection model for de-reflection processing, and obtain multiple de-reflection reconstructed object images; wherein the de-reflection model sequentially downsamples and upsamples the input image multiple times, and fuses the result of each upsampling and the input image through channel splicing and transposed convolution to obtain a de-reflection image; Using a pre-trained de-occlusion model, based on an occlusion mask map of the occlusion area in each de-reflection reconstructed object image, de-occlusion processing is performed on the occlusion area of ​​each de-reflection reconstructed object image to obtain a plurality of processed reconstructed object images; Performing three-dimensional reconstruction using each of the processed reconstructed object images to obtain a three-dimensional model of the object to be reconstructed; The de-occlusion process includes: Inputting the de-reflected reconstructed object image and the corresponding occlusion mask map into the de-occlusion network in the de-occlusion model, and performing convolution respectively through the multiple convolution layers in the de-occlusion network to obtain a convolution feature map; The convolution feature map is extracted and fused sequentially through the three FFC modules in the de-occlusion network; The output of the last FFC module is combined with the convolutional feature map, and processed through multiple convolutional layers in the de-occlusion network to obtain the processed reconstructed object image.

2. The method according to claim 1, characterized in that: The multi-directional image acquisition of the object to be reconstructed to obtain multiple images of the object to be reconstructed includes: At a plurality of positions distributed around the object to be reconstructed, images of the object to be reconstructed corresponding to the positions are acquired; wherein the images of the object to be reconstructed corresponding to the plurality of positions include images acquired from a bird's-eye view, images acquired from a level view, and images acquired from an upward view.

3. The method according to claim 1, characterized in that: The step of inputting each of the to-be-reconstructed object images into a pre-trained de-reflection model for de-reflection processing to obtain a plurality of de-reflection reconstructed object images comprises: For each of the to-be-reconstructed object images, the to-be-reconstructed object images are input into a pre-trained de-reflection model, down-sampled sequentially through each convolution layer in the encoder of the de-reflection model, and up-sampled sequentially through multiple convolution layers in the decoder to obtain a convolution result; The convolution result, the input image and the final transposed convolution result are channel-stitched to obtain the de-reflected reconstructed object image; wherein the final transposed convolution result is obtained by sequentially channel-stitching the convolution results of each convolution layer of the decoder with the current transposed convolution result, and transposing the convolution result.

4. The method according to claim 3, characterized in that The training method of the de-reflection model comprises: Capturing a color image of a sample object separated by a transparent material to obtain a first sample image, and capturing an image of a sample object not separated by a transparent material to obtain a first label image; Inputting the first sample image into the de-reflection model, and processing the first sample image by the de-reflection model to obtain a current de-reflection image; Calculate the error between the first label image and the current de-reflection image to obtain a current output result loss value, and calculate the error between the first label image and each transposed convolution result to obtain each current transposed convolution result loss value; wherein the transposed convolution result includes the result obtained by transposing the convolution result of the last convolution layer of the encoder in the de-reflection model, and the result of the transposed convolution of the current layer obtained by performing channel splicing and transposing convolution on each convolution layer except the last layer in the decoder in the de-reflection model with the transposed convolution result of the previous layer; Weighting the current output result loss value and each current transposed convolution result loss value to obtain the loss value of the current de-reflection model; Determine whether the loss value of the current de-reflection model converges; If it is determined that the loss value of the current de-reflection model has not converged, the parameters of the de-reflection model are adjusted, and the step of inputting the first sample image into the de-reflection model is returned to execute, and the first sample image is processed by the de-reflection model to obtain a current de-reflection image; If it is determined that the loss value of the current de-reflection model has converged, the training of the de-reflection model is terminated.

5. The method according to claim 1, characterized in that: The process of generating an occlusion mask map of an occlusion area in a de-reflection reconstructed object image includes: For each of the de-reflection reconstructed object images, the de-reflection reconstructed object images are input into the de-occlusion model, the occlusion regions in the de-reflection reconstructed object images are detected by an occlusion region detection network in the de-occlusion model, and an occlusion mask map of the occlusion regions in the de-reflection reconstructed object images is generated; The de-occlusion model is trained in advance using images with and without occlusion areas, and a mask image of the existing occlusion areas.

6. The method according to claim 1, characterized in that The training method of the de-occlusion model comprises: Collecting an original image of the object, and setting a blocking area on the original image of the object to obtain a second sample image; Create a mask image of the set occluded area to obtain a second label image; Iteratively training the occlusion detection network in the de-occlusion model using the second sample image and the second label image until the error between the current mask image output by the occlusion detection network and the second label image converges; The original image of the object is used as the third label image, and the deocclusion network in the deocclusion model is iteratively trained using the second sample image and the current mask image output by the occlusion detection network until the error between the deoccluded image output by the deocclusion network and the third label image converges.

7. A three-dimensional model reconstruction device, characterized in that: include: An image acquisition unit, used for acquiring multi-directional images of the object to be reconstructed to obtain multiple images of the object to be reconstructed; A dereflection unit, used for inputting each of the images of the object to be reconstructed into a pre-trained dereflection model for dereflection processing, so as to obtain a plurality of dereflection reconstructed object images; wherein the dereflection model sequentially downsamples and upsamples the input image multiple times, and fuses the result of each upsampling and the input image through channel splicing and transposed convolution, so as to obtain a dereflected image; a de-occlusion unit, configured to perform de-occlusion processing on the occluded area of ​​each de-reflected reconstructed object image based on an occlusion mask map of the occluded area in each de-reflected reconstructed object image by using a pre-trained de-occlusion model, so as to obtain a plurality of processed reconstructed object images; A reconstruction unit, used for performing three-dimensional reconstruction using each of the processed reconstructed object images to obtain a three-dimensional model of the object to be reconstructed; When the de-occlusion unit performs the de-occlusion processing, it is specifically used to: Inputting the de-reflected reconstructed object image and the corresponding occlusion mask map into the de-occlusion network in the de-occlusion model, and performing convolution respectively through the multiple convolution layers in the de-occlusion network to obtain a convolution feature map; The convolution feature map is extracted and fused sequentially through the three FFC modules in the de-occlusion network; The output of the last FFC module is combined with the convolutional feature map, and processed through multiple convolutional layers in the de-occlusion network to obtain the processed reconstructed object image.

8. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the three-dimensional model reconstruction method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that Used to store a computer program, which, when executed, is used to implement the three-dimensional model reconstruction method as described in any one of claims 1 to 6.

Images