Cultural relic three-dimensional modeling method coupling DUSt3R and U-Net semantic segmentation

Abstract

The invention provides a DUSt3R and U-Net semantic segmentation coupled cultural relic three-dimensional modeling method, and the method comprises the steps: obtaining a small number of original images of a target cultural relic shot by a tourist in real time; preprocessing a small number of original images of the target cultural relic to obtain a small number of images of the target cultural relic; inputting the small number of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a target cultural relic three-dimensional model corresponding to the small number of images of the target cultural relic output by the three-dimensional reconstruction model; wherein the three-dimensional reconstruction model is obtained by training a sample image of the sample cultural relic and a sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic. According to the method, the three-dimensional reconstruction process can be simplified, the calculation complexity is reduced, the three-dimensional reconstruction efficiency and real-time performance are improved, and the tourist experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a cultural relic three-dimensional modeling method coupling DUSt3R and U-Net semantic segmentation. Background Art

[0002] With the rapid development of 3D reconstruction technology, its application in the field of cultural relics protection is becoming more and more extensive. Museums usually provide virtual reality 3D models of cultural relics for tourists to visit. However, although this method provides convenience, tourists' sense of participation is poor. In the scenario of tourists taking photos, how to use the photos of cultural relics taken by tourists to generate 3D models of cultural relics in real time to enhance tourists' sense of participation in appreciating and protecting cultural relics is a key issue that needs to be solved urgently.

[0003] At present, the existing technology uses a combination of three-dimensional laser scanning and multi-view photography to reconstruct the images of cultural relics in three dimensions. This not only requires the collection of cultural relic images from multiple projection perspectives, but also requires the calibration of camera parameters. The complexity is high and the three-dimensional reconstruction efficiency is low. It is difficult to use the limited number of cultural relic photos taken by tourists to generate a three-dimensional model of cultural relics in real time, which affects the tourists' experience. Summary of the invention

[0004] The present invention provides a cultural relic 3D modeling method coupled with DUSt3R and U-Net semantic segmentation, so as to solve the problem that in the prior art, a technology combining 3D laser scanning and multi-view photography is used to perform 3D reconstruction of cultural relic images, which not only requires the collection of cultural relic images from multiple projection viewpoints, but also requires the calibration of camera parameters, resulting in high complexity and low 3D reconstruction efficiency. It is difficult to generate a 3D model of cultural relics in real time using a limited number of cultural relic photos taken by tourists, thus affecting the tourists' experience.

[0005] In a first aspect, the present invention provides a method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation, comprising: Acquire a small number of original images of target cultural relics taken by tourists in real time; Preprocessing a small number of original images of the target cultural relics to obtain a small number of images of the target cultural relics; Inputting a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; The three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

[0006] In some embodiments, the 3D reconstruction model includes a semantic segmentation layer and a 3D reconstruction layer, the semantic segmentation layer includes a U-Net network, and the 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model.

[0007] In some embodiments, the step of inputting a small number of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small number of images of the target cultural relic output by the three-dimensional reconstruction model comprises: Inputting a small amount of images of the target cultural relics into the semantic segmentation layer to obtain a small amount of segmentation maps of the target cultural relics output by the semantic segmentation layer; A small amount of segmentation images of the target cultural relics are input into the three-dimensional reconstruction layer to obtain a three-dimensional model of the target cultural relics corresponding to a small amount of images of the target cultural relics output by the three-dimensional reconstruction layer.

[0008] In some embodiments, the step of inputting a small number of images of the target cultural relics into the semantic segmentation layer to obtain a small number of segmentation maps of the target cultural relics output by the semantic segmentation layer includes: Based on the semantic segmentation layer, semantic segmentation is performed on a small number of images of the target cultural relics to obtain a small number of mask images of the target cultural relics; A small number of mask images of the target cultural relics are post-processed to obtain a small number of segmentation images of the target cultural relics.

[0009] In some embodiments, the step of inputting a small number of segmentation images of the target cultural relic into the 3D reconstruction layer to obtain a 3D model of the target cultural relic corresponding to a small number of images of the target cultural relic output by the 3D reconstruction layer includes: Based on the three-dimensional reconstruction layer, globally aligning a small number of segmentation maps of the target cultural relics to obtain aligned segmentation maps; Three-dimensional reconstruction is performed based on the aligned segmentation map to obtain a three-dimensional model of the target cultural relic corresponding to a small number of images of the target cultural relic.

[0010] In some embodiments, the process of determining the three-dimensional reconstruction model includes: Obtaining sample original images of sample artifacts; Preprocessing the sample original image of the sample cultural relic to obtain a sample image of the sample cultural relic; Determine a sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Using the sample images of the sample cultural relics as training samples and the sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics as sample labels, training an initial three-dimensional reconstruction model; The parameters of the initial three-dimensional reconstruction model are iteratively optimized, and after training is completed, the three-dimensional reconstruction model is obtained.

[0011] In some embodiments, the initial 3D reconstruction model includes an initial semantic segmentation layer and an initial 3D reconstruction layer, the initial semantic segmentation layer includes a U-Net network, and the initial 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model; Correspondingly, the training of the initial three-dimensional reconstruction model includes: Inputting the sample image of the sample cultural relic into the initial semantic segmentation layer to obtain a sample segmentation map of the sample cultural relic output by the initial semantic segmentation layer; Inputting the sample segmentation map of the sample cultural relic into the initial three-dimensional reconstruction layer to obtain a three-dimensional model of the sample cultural relic corresponding to the sample image of the sample cultural relic output by the initial three-dimensional reconstruction layer; Calculating a loss function value based on the sample cultural relic three-dimensional model corresponding to the sample image of the sample cultural relic and the sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Based on the loss function value, parameters of the initial semantic segmentation layer and the initial three-dimensional reconstruction layer are iteratively optimized.

[0012] In a second aspect, the present invention further provides a cultural relic 3D modeling device coupled with DUSt3R and U-Net semantic segmentation, comprising: An acquisition unit, used for acquiring a small amount of original images of target cultural relics photographed by tourists in real time; A preprocessing unit, used for preprocessing a small amount of original images of the target cultural relics to obtain a small amount of images of the target cultural relics; A three-dimensional reconstruction unit, used for inputting a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model, and obtaining a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; The three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

[0013] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for three-dimensional modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation as described in any one of the above-mentioned methods is implemented.

[0014] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for three-dimensional modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation as described in any one of the above.

[0015] The cultural relic 3D modeling method coupled with DUSt3R and U-Net semantic segmentation provided by the present invention obtains a small number of original images of target cultural relics taken by tourists in real time, pre-processes the small number of original images of target cultural relics to obtain a small number of images of target cultural relics, inputs the small number of images of target cultural relics into a pre-trained 3D reconstruction model, and obtains a 3D model of the target cultural relic corresponding to the small number of images of target cultural relics output by the 3D reconstruction model. The 3D reconstruction process can be simplified, the complexity of calculation can be reduced, the efficiency and real-time performance of 3D reconstruction can be improved, and the tourist experience can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation provided by an embodiment of the present invention; Figure 2 is a schematic flow chart of a process for determining a three-dimensional reconstruction model provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a cultural relic three-dimensional modeling device for coupling DUSt3R and U-Net semantic segmentation provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, the 3D reconstruction technology for cultural relics protection and digital presentation has developed to a relatively mature level. Museums usually use a combination of 3D laser scanning and multi-view photography to reconstruct cultural relics in 3D, and the reconstruction results are highly accurate. However, 3D laser scanning instruments are large and expensive. They need to take and measure photos of cultural relics from hundreds of projection perspectives, and establish corresponding camera imaging geometric models (i.e., camera parameter calibration). The data processing of the entire process takes a long time. Tourists can generally only use uncalibrated mobile phones or SLR cameras to avoid being blocked by other tourists around the cultural relics and quickly take two or three photos from a few limited perspectives. The number of photos is extremely small. Moreover, these shooting perspectives with unknown coordinates are difficult to be accurately restored in the reconstruction work. Therefore, the 3D reconstruction technology currently used by museums cannot be applied to personalized 3D modeling in tourist photo scenes.

[0020] In addition, in the scenes of cultural relics photographed by tourists, there are inevitably background objects that are irrelevant to the 3D reconstruction of cultural relics, such as other passers-by, museum display cabinets, etc. If all objects in the scene are directly reconstructed, the subjectivity of the reconstruction of the cultural relics cannot be highlighted. How to accurately segment the target cultural relics to be reconstructed from the cluttered scenes is one of the key issues to be solved.

[0021] To this end, the present invention provides a cultural relic 3D modeling method coupled with DUSt3R and U-Net semantic segmentation, by real-time acquisition of a small number of original images of target cultural relics taken by tourists, preprocessing the small number of original images of target cultural relics, obtaining a small number of images of target cultural relics, inputting the small number of images of target cultural relics into a pre-trained 3D reconstruction model, and obtaining a 3D model of the target cultural relic corresponding to the small number of images of target cultural relics output by the 3D reconstruction model. The present invention can simplify the process of 3D reconstruction of cultural relics, reduce the complexity of calculation, improve the efficiency and real-time performance of 3D reconstruction, and enhance the tourist experience.

[0022] Figure 1 A flow chart of a method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation provided in an embodiment of the present invention. Figure 1 As shown, a method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation is provided, comprising the following steps: step 110, step 120, and step 130. The steps of the method flow are only a possible implementation of the present invention.

[0023] Step 110: Acquire a small amount of original images of the target cultural relics photographed by tourists in real time.

[0024] The original images include two-dimensional images of target artifacts and scenes, and a small number of original images refers to more than two original images.

[0025] Optionally, the target artifact is a single small asymmetric artifact indoors.

[0026] Optionally, obtain 2 to 4 original images of the target cultural relic taken by tourists using a mobile phone or camera.

[0027] Step 120: pre-process a small number of original images of the target cultural relics to obtain a small number of images of the target cultural relics.

[0028] Optionally, a small amount of original images of the target cultural relics are preprocessed by denoising, image enhancement, etc.

[0029] Optionally, a small number of original images of the target cultural relics are screened, and original images in which the cultural relics are severely obscured are eliminated to obtain a small number of images of the target cultural relics.

[0030] Step 130: input a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; The three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

[0031] In some embodiments, the 3D reconstruction model includes a semantic segmentation layer and a 3D reconstruction layer, the semantic segmentation layer includes a U-Net network, and the 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model.

[0032] Among them, the U-Net network includes an encoder and a decoder. The U-Net network can not only effectively capture the global context information in the image and perform efficient feature extraction, but also retain spatial information through jump connections, thereby achieving high-precision pixel-level classification in image segmentation tasks.

[0033] Among them, the Dense and Unconstrained Stereo 3D Reconstruction (DUSt3R) model includes a Transformer encoder and decoder. DUSt3R aims to simplify the process of reconstructing 3D scenes from any image collection without the need for camera calibration or prior knowledge of viewpoint position information. DUSt3R relaxes the constraints of the traditional projective camera model by treating the paired reconstruction problem as a regression problem of point graphs, and can learn rich geometric and appearance information from the input image without explicit geometric constraints.

[0034] It is understandable that by adopting the 3D reconstruction model coupled with DUSt3R and U-Net networks, it is possible to obtain the corresponding 3D model of the target cultural relic based on a small number of input images of the target cultural relic, without the need for camera calibration and perspective positioning, thereby simplifying the process of 3D reconstruction of cultural relics, getting rid of the dependence of cultural relic reconstruction on expensive equipment and numerous perspective images, reducing the complexity of data processing, and lowering the threshold for 3D reconstruction of cultural relics, so that ordinary tourists can also reconstruct their own 3D models of cultural relics from photos taken with mobile phones, which helps to enhance tourists' viewing pleasure and promote the dissemination of the traditional cultural essence contained in cultural relics.

[0035] In an embodiment of the present invention, by acquiring a small number of original images of target cultural relics taken by tourists in real time, preprocessing the small number of original images of target cultural relics to obtain a small number of images of target cultural relics, inputting the small number of images of target cultural relics into a pre-trained 3D reconstruction model, and obtaining a 3D model of the target cultural relic corresponding to the small number of images of target cultural relics output by the 3D reconstruction model, the 3D reconstruction process can be simplified, the complexity of calculation can be reduced, the efficiency and real-time performance of 3D reconstruction can be improved, and the tourist experience can be enhanced.

[0036] In some embodiments, a small number of images of a target cultural relic are input into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small number of images of the target cultural relic output by the three-dimensional reconstruction model, including: Input a small number of images of the target cultural relics into the semantic segmentation layer, and obtain a small number of segmentation maps of the target cultural relics output by the semantic segmentation layer; A small amount of segmentation maps of the target cultural relics are input into the three-dimensional reconstruction layer, and a three-dimensional model of the target cultural relics corresponding to a small amount of images of the target cultural relics output by the three-dimensional reconstruction layer is obtained.

[0037] Optionally, a small number of segmentation images of the target cultural relic are input into the 3D reconstruction layer to obtain a 3D model of the target cultural relic corresponding to a small number of images of the target cultural relic output by the 3D reconstruction layer, as well as a depth map and a confidence map corresponding to each image of the target cultural relic.

[0038] It can be understood that by inputting a small number of images of the target cultural relics into the semantic segmentation layer and obtaining a small number of segmentation maps of the target cultural relics output by the semantic segmentation layer, the cultural relic area and the background area of ​​the input image can be accurately divided, providing accurate basic data for subsequent three-dimensional reconstruction and improving the efficiency of three-dimensional reconstruction.

[0039] In some embodiments, a small number of images of the target cultural relics are input into the semantic segmentation layer, and a small number of segmentation maps of the target cultural relics output by the semantic segmentation layer are obtained, including: Based on the semantic segmentation layer, a small number of images of the target cultural relics are semantically segmented to obtain a small number of mask images of the target cultural relics; A small number of mask images of the target cultural relics are post-processed to obtain a small number of segmentation images of the target cultural relics.

[0040] The mask image includes a black background and a white target artifact shape; the segmentation image includes the target artifact image after removing the background.

[0041] Optionally, binary classification is performed on the artifact region and the background region in a small number of images of the target artifact.

[0042] Optionally, background areas in a small number of images of the target cultural relics are blocked to obtain a small number of mask images of the target cultural relics.

[0043] Optionally, a small number of mask images of the target cultural relics are post-processed by background filtering, background color conversion, etc. to obtain a small number of segmentation images of the target cultural relics.

[0044] In some embodiments, a small number of segmentation images of the target cultural relic are input into the 3D reconstruction layer to obtain a 3D model of the target cultural relic corresponding to a small number of images of the target cultural relic output by the 3D reconstruction layer, including: Based on the 3D reconstruction layer, a small number of segmentation maps of the target cultural relics are globally aligned to obtain aligned segmentation maps; Three-dimensional reconstruction is performed based on the aligned segmentation map to obtain a three-dimensional model of the target cultural relic corresponding to a small number of images of the target cultural relic.

[0045] Optionally, the points of a small number of segmentation maps of the target cultural relics are globally aligned, and the alignment parameters are adjusted using an optimization algorithm to obtain the aligned segmentation maps.

[0046] Optionally, based on the aligned segmentation map, point cloud data of the target cultural relic is generated, and the point cloud data is converted into a triangular mesh model to form an initial three-dimensional model; the original image texture of the target cultural relic is mapped onto the initial three-dimensional model to obtain the final three-dimensional model of the target cultural relic.

[0047] Figure 2 A schematic diagram of a process for determining a three-dimensional reconstruction model according to an embodiment of the present invention is shown in FIG. Figure 2 As shown, in some embodiments, the process of determining the three-dimensional reconstruction model includes: Step 210, obtaining a sample original image of the sample cultural relic; Step 220, preprocessing the sample original image of the sample cultural relic to obtain a sample image of the sample cultural relic; Step 230, determining a sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Step 240: using the sample images of the sample cultural relics as training samples and the sample cultural relic 3D model labels corresponding to the sample images of the sample cultural relics as sample labels to train an initial 3D reconstruction model; Step 250: Iteratively optimize the parameters of the initial three-dimensional reconstruction model. After the training is completed, a three-dimensional reconstruction model is obtained.

[0048] Optionally, sample original images of a plurality of sample cultural relics of different types and in different states are taken by a mobile phone or a camera.

[0049] Optionally, use the open source annotation tool Labelme to annotate the sample original images of the sample artifacts.

[0050] Optionally, a training data set and a validation data set are constructed based on sample images of sample cultural relics and labels of three-dimensional models of sample cultural relics corresponding to the sample images of sample cultural relics.

[0051] Optionally, the three-dimensional reconstruction model obtained after training is verified according to the verification data set, and the parameters of the three-dimensional reconstruction model are adjusted according to the verification result.

[0052] In some embodiments, the initial 3D reconstruction model includes an initial semantic segmentation layer and an initial 3D reconstruction layer, the initial semantic segmentation layer includes a U-Net network, and the initial 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model; Correspondingly, training the initial 3D reconstruction model includes: Inputting a sample image of the sample cultural relic into an initial semantic segmentation layer to obtain a sample segmentation map of the sample cultural relic output by the initial semantic segmentation layer; Inputting the sample segmentation map of the sample cultural relic into the initial three-dimensional reconstruction layer to obtain a three-dimensional model of the sample cultural relic corresponding to the sample image of the sample cultural relic output by the initial three-dimensional reconstruction layer; Calculate the loss function value based on the sample cultural relic three-dimensional model corresponding to the sample image of the sample cultural relic and the sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Based on the loss function value, the parameters of the initial semantic segmentation layer and the initial 3D reconstruction layer are iteratively optimized.

[0053] Optionally, the Dice coefficient is used as a quantitative evaluation index of the semantic segmentation layer to test the semantic segmentation layer.

[0054] Optionally, the original image of the target cultural relic is compared with the corresponding mask image to determine the accuracy and missing content of the mask image obtained after semantic segmentation, and the parameters of the semantic segmentation layer are optimized.

[0055] Optionally, images of multiple cultural relics are randomly selected, and three-dimensional reconstruction of the multiple cultural relics is performed in combination with the trained three-dimensional reconstruction model, the intuitive reconstruction effects of different cultural relics are compared, and the parameters of the three-dimensional reconstruction model are optimized.

[0056] It can be understood that by using the sample images of sample cultural relics as training samples, the sample cultural relic three-dimensional model labels corresponding to the sample images of sample cultural relics as sample labels, training the initial three-dimensional reconstruction model, and iteratively optimizing the parameters of the initial three-dimensional reconstruction model to obtain the three-dimensional reconstruction model, the details of the sample cultural relics can be more effectively captured, thereby improving the accuracy and reliability of the three-dimensional reconstruction.

[0057] The following is a description of the cultural relics three-dimensional modeling device for coupling DUSt3R and U-Net semantic segmentation provided by an embodiment of the present invention. The cultural relics three-dimensional modeling device for coupling DUSt3R and U-Net semantic segmentation described below and the cultural relics three-dimensional modeling method for coupling DUSt3R and U-Net semantic segmentation described above can be referenced to each other.

[0058] Figure 3 A schematic diagram of the structure of a cultural relic 3D modeling device for coupling DUSt3R and U-Net semantic segmentation provided in an embodiment of the present invention, as shown in FIG. Figure 3 As shown, the cultural relics 3D modeling device 300 coupled with DUSt3R and U-Net semantic segmentation includes: An acquisition unit 310 is used to acquire a small amount of original images of target cultural relics photographed by tourists in real time; A preprocessing unit 320 is used to preprocess a small amount of original images of the target cultural relics to obtain a small amount of images of the target cultural relics; The three-dimensional reconstruction unit 330 is used to input a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; The three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

[0059] Optionally, the 3D reconstruction model includes a semantic segmentation layer and a 3D reconstruction layer, the semantic segmentation layer includes a U-Net network, and the 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model.

[0060] Optionally, a small number of images of the target cultural relic are input into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small number of images of the target cultural relic output by the three-dimensional reconstruction model, including: Input a small number of images of the target cultural relics into the semantic segmentation layer, and obtain a small number of segmentation maps of the target cultural relics output by the semantic segmentation layer; A small amount of segmentation maps of the target cultural relics are input into the three-dimensional reconstruction layer, and a three-dimensional model of the target cultural relics corresponding to a small amount of images of the target cultural relics output by the three-dimensional reconstruction layer is obtained.

[0061] Optionally, a small number of images of the target cultural relics are input into the semantic segmentation layer, and a small number of segmentation maps of the target cultural relics output by the semantic segmentation layer are obtained, including: Based on the semantic segmentation layer, a small number of images of the target cultural relics are semantically segmented to obtain a small number of mask images of the target cultural relics; A small number of mask images of the target cultural relics are post-processed to obtain a small number of segmentation images of the target cultural relics.

[0062] Optionally, a small number of segmentation images of the target cultural relics are input into the three-dimensional reconstruction layer to obtain a three-dimensional model of the target cultural relics corresponding to the small number of images of the target cultural relics output by the three-dimensional reconstruction layer, including: Based on the 3D reconstruction layer, a small number of segmentation maps of the target cultural relics are globally aligned to obtain aligned segmentation maps; Three-dimensional reconstruction is performed based on the aligned segmentation map to obtain a three-dimensional model of the target cultural relic corresponding to a small number of images of the target cultural relic.

[0063] Optionally, the process of determining the three-dimensional reconstruction model includes: Obtaining sample original images of sample artifacts; Preprocessing the sample original image of the sample cultural relic to obtain a sample image of the sample cultural relic; Determine a sample cultural relic three-dimensional model label corresponding to a sample image of the sample cultural relic; Using sample images of sample cultural relics as training samples and using sample cultural relic 3D model labels corresponding to the sample images of sample cultural relics as sample labels, training an initial 3D reconstruction model; The parameters of the initial 3D reconstruction model are iteratively optimized, and after training is completed, a 3D reconstruction model is obtained.

[0064] Optionally, the initial 3D reconstruction model includes an initial semantic segmentation layer and an initial 3D reconstruction layer, the initial semantic segmentation layer includes a U-Net network, and the initial 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model; Correspondingly, training the initial 3D reconstruction model includes: Inputting a sample image of the sample cultural relic into an initial semantic segmentation layer to obtain a sample segmentation map of the sample cultural relic output by the initial semantic segmentation layer; Inputting the sample segmentation map of the sample cultural relic into the initial three-dimensional reconstruction layer to obtain a three-dimensional model of the sample cultural relic corresponding to the sample image of the sample cultural relic output by the initial three-dimensional reconstruction layer; Calculate the loss function value based on the sample cultural relic three-dimensional model corresponding to the sample image of the sample cultural relic and the sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Based on the loss function value, the parameters of the initial semantic segmentation layer and the initial 3D reconstruction layer are iteratively optimized.

[0065] It should be noted here that the cultural relics three-dimensional modeling device coupled with DUSt3R and U-Net semantic segmentation provided in an embodiment of the present invention can implement all the method steps implemented in the above-mentioned cultural relics three-dimensional modeling method embodiment coupled with DUSt3R and U-Net semantic segmentation, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0066] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the cultural relic three-dimensional modeling method coupled with DUSt3R and U-Net semantic segmentation, the method comprising: acquiring a small amount of original images of the target cultural relic taken by tourists in real time; preprocessing the small amount of original images of the target cultural relic to obtain a small amount of images of the target cultural relic; inputting the small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a target cultural relic three-dimensional model corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; wherein the three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

[0067] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cultural relic three-dimensional modeling method coupled with DUSt3R and U-Net semantic segmentation provided by the above-mentioned methods, the method comprising: acquiring a small number of original images of target cultural relics taken by tourists in real time; preprocessing the small number of original images of the target cultural relics to obtain a small number of images of the target cultural relics; inputting the small number of images of the target cultural relics into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relics corresponding to the small number of images of the target cultural relics output by the three-dimensional reconstruction model; wherein the three-dimensional reconstruction model is trained based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation, characterized in that: include: Acquire a small number of original images of target cultural relics taken by tourists in real time; Preprocessing a small number of original images of the target cultural relics to obtain a small number of images of the target cultural relics; Inputting a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; The three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

2. The method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation according to claim 1, characterized in that: The three-dimensional reconstruction model includes a semantic segmentation layer and a three-dimensional reconstruction layer, the semantic segmentation layer includes a U-Net network, and the three-dimensional reconstruction layer includes a dense unconstrained stereo three-dimensional reconstruction DUSt3R model.

3. The method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation according to claim 2, characterized in that: The step of inputting a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model to obtain a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model comprises: Inputting a small amount of images of the target cultural relics into the semantic segmentation layer to obtain a small amount of segmentation maps of the target cultural relics output by the semantic segmentation layer; A small amount of segmentation images of the target cultural relics are input into the three-dimensional reconstruction layer to obtain a three-dimensional model of the target cultural relics corresponding to a small amount of images of the target cultural relics output by the three-dimensional reconstruction layer.

4. The method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation according to claim 3, characterized in that: The step of inputting a small amount of images of the target cultural relics into the semantic segmentation layer to obtain a small amount of segmentation maps of the target cultural relics output by the semantic segmentation layer comprises: Based on the semantic segmentation layer, semantic segmentation is performed on a small number of images of the target cultural relics to obtain a small number of mask images of the target cultural relics; A small number of mask images of the target cultural relics are post-processed to obtain a small number of segmentation images of the target cultural relics.

5. The method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation according to claim 3, characterized in that: The step of inputting a small amount of segmentation images of the target cultural relics into the three-dimensional reconstruction layer to obtain a three-dimensional model of the target cultural relics corresponding to a small amount of images of the target cultural relics output by the three-dimensional reconstruction layer comprises: Based on the three-dimensional reconstruction layer, globally aligning a small number of segmentation maps of the target cultural relics to obtain aligned segmentation maps; Three-dimensional reconstruction is performed based on the aligned segmentation map to obtain a three-dimensional model of the target cultural relic corresponding to a small number of images of the target cultural relic.

6. The method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation according to claim 1, characterized in that: The determination process of the three-dimensional reconstruction model includes: Obtaining sample original images of sample artifacts; Preprocessing the sample original image of the sample cultural relic to obtain a sample image of the sample cultural relic; Determine a sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Using the sample images of the sample cultural relics as training samples and the sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics as sample labels, training an initial three-dimensional reconstruction model; The parameters of the initial three-dimensional reconstruction model are iteratively optimized, and after training is completed, the three-dimensional reconstruction model is obtained.

7. The method for 3D modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation according to claim 6, characterized in that: The initial 3D reconstruction model includes an initial semantic segmentation layer and an initial 3D reconstruction layer, the initial semantic segmentation layer includes a U-Net network, and the initial 3D reconstruction layer includes a dense unconstrained stereo 3D reconstruction DUSt3R model; Correspondingly, the training of the initial three-dimensional reconstruction model includes: Inputting the sample image of the sample cultural relic into the initial semantic segmentation layer to obtain a sample segmentation map of the sample cultural relic output by the initial semantic segmentation layer; Inputting the sample segmentation map of the sample cultural relic into the initial three-dimensional reconstruction layer to obtain a three-dimensional model of the sample cultural relic corresponding to the sample image of the sample cultural relic output by the initial three-dimensional reconstruction layer; Calculating a loss function value based on the sample cultural relic three-dimensional model corresponding to the sample image of the sample cultural relic and the sample cultural relic three-dimensional model label corresponding to the sample image of the sample cultural relic; Based on the loss function value, parameters of the initial semantic segmentation layer and the initial three-dimensional reconstruction layer are iteratively optimized.

8. A 3D modeling device for cultural relics by coupling DUSt3R and U-Net semantic segmentation, characterized in that: include: An acquisition unit, used for acquiring a small amount of original images of target cultural relics photographed by tourists in real time; A preprocessing unit, used for preprocessing a small amount of original images of the target cultural relics to obtain a small amount of images of the target cultural relics; A three-dimensional reconstruction unit, used for inputting a small amount of images of the target cultural relic into a pre-constructed three-dimensional reconstruction model, and obtaining a three-dimensional model of the target cultural relic corresponding to the small amount of images of the target cultural relic output by the three-dimensional reconstruction model; The three-dimensional reconstruction model is obtained by training based on sample images of sample cultural relics and sample cultural relic three-dimensional model labels corresponding to the sample images of the sample cultural relics.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for three-dimensional modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation as described in any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for three-dimensional modeling of cultural relics by coupling DUSt3R and U-Net semantic segmentation as described in any one of claims 1 to 6 is implemented.

Images