Three-dimensional reconstruction method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method, electronic equipment and a storage medium. The method comprises the steps of determining features of a preset object according to initial scanning data of the preset object; determining target scanning modes corresponding to different areas of the preset object according to the features; responding to a target scanning mode selected by a user, starting the selected target scanning mode to scan a corresponding area of the preset object, and obtaining scanning data of each area of the preset object; and fusing the scanning data of all the areas of the preset object to obtain a three-dimensional reconstruction model of the preset object. By means of the method, the efficiency and precision of three-dimensional reconstruction can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional reconstruction, and in particular, to a three-dimensional reconstruction method, an electronic device and a storage medium. Background Art

[0002] Existing 3D scanning technology faces great limitations in scanning fixed scenes, resulting in poor results for 3D reconstructed models. For example, when using an ordinary camera, in order to see the entire object clearly, it can only be shot from a distance, resulting in blurred details in the scanned data. In addition, when scanning object areas with complex structures (such as grooves), it is necessary to scan each small area closely, and the position must be adjusted repeatedly to complete the splicing, which is not only time-consuming, but the errors from multiple scans will also be superimposed like the misalignment of puzzle pieces, resulting in deformation of the final model. Summary of the invention

[0003] The present application discloses a three-dimensional reconstruction method, an electronic device and a storage medium, which can switch the scanning mode according to the regional characteristics of the scanning object, improve the scanning efficiency while retaining sufficient scanning details, and solve the contradiction between scanning efficiency and detail retention in a fixed scanning scene.

[0004] A first aspect of an embodiment of the present application provides a three-dimensional reconstruction method, comprising: determining features of a preset object based on initial scanning data of the preset object; determining target scanning modes corresponding to different areas of the preset object based on the features; in response to a target scanning mode selected by a user, starting the selected target scanning mode to scan the corresponding area of ​​the preset object to obtain scanning data of each area of ​​the preset object; and fusing the scanning data of all areas of the preset object to obtain a three-dimensional reconstructed model of the preset object.

[0005] According to an embodiment of the present application, before determining the characteristics of the preset object based on the scanning data of the preset object, the method also includes: in response to the scanning start operation performed by the user, scanning the preset object using an initial scanning mode to obtain the initial scanning data, wherein the initial scanning mode includes a monocular scanning mode and a binocular scanning mode.

[0006] According to an embodiment of the present application, the initial scanning data includes image data, and determining the features of the preset object based on the initial scanning data of the preset object includes: performing semantic segmentation on the preset object based on the image data, and determining multiple different areas of the preset object based on the results of the semantic segmentation; determining the feature point density and depth change rate of each area of ​​the preset object, and determining the features based on the feature point density and the depth change rate.

[0007] According to an embodiment of the present application, the target scanning mode includes at least one of the following: a binocular scanning mode in a first range, a monocular and binocular combination mode in a first range, a binocular scanning mode in a second range, and a monocular and binocular combination mode in a second range, wherein the first range is larger than the second range, and the monocular and binocular combination mode represents a combination mode of a monocular scanning mode and a binocular scanning mode.

[0008] According to an embodiment of the present application, determining the target scanning mode corresponding to different areas of the preset object according to the features includes: dividing the different areas of the preset object into multiple categories of areas according to the features, the multiple categories of areas including first category areas, second category areas, third category areas and fourth category areas; determining the target scanning mode corresponding to the first category of areas as the binocular scanning mode of the first range; determining the target scanning mode corresponding to the second category of areas as the binocular scanning mode of the second range; determining the target scanning mode corresponding to the third category of areas as the single and binocular combination mode of the first range; determining the target scanning mode corresponding to the fourth category of areas as the single and binocular combination mode of the second range.

[0009] According to an embodiment of the present application, the first type of area represents an area whose geometric area is greater than a preset area threshold, the feature point density is less than or equal to the preset density threshold, and the depth change rate is less than or equal to the preset change rate threshold; the second type of area represents an area whose geometric area is less than or equal to the preset area threshold, the feature point density is greater than the preset density threshold, and the depth change rate is less than or equal to the preset change rate threshold; the third type of area represents an area whose geometric area is greater than the preset area threshold and the depth change rate is greater than the preset change rate threshold; the fourth type of area represents an area whose geometric area is less than or equal to the preset area threshold and the depth change rate is greater than the preset change rate threshold.

[0010] According to an embodiment of the present application, the scanning data of all areas of the preset object are fused to obtain a three-dimensional reconstructed model of the preset object, including: if the scanning data includes monocular scanning data and binocular scanning data, coordinate aligning the monocular scanning data with the binocular scanning data; determining the overlapping area of ​​the monocular scanning data and the binocular scanning data after the coordinate alignment, and removing redundant data in the overlapping area; performing global data fusion based on the scanning data obtained after removing redundant data from all areas to obtain global fused data; and globally optimizing the global fused data to obtain the three-dimensional reconstructed model.

[0011] According to an embodiment of the present application, the scan data obtained after removing redundant data from all areas is subjected to global data fusion to obtain global fused data, including: according to the positions of preset landmark points of the preset objects, the scan data after removing redundant data from all areas is globally spliced ​​to obtain the global fused data.

[0012] According to an embodiment of the present application, the optimization of the global fusion data includes: performing a de-overlapping operation on the global fusion data and retaining the binocular scanning data in the de-overlapping operation; using a uniform point distance transition algorithm to perform smooth transition processing on the edges of the scanning data in different areas.

[0013] According to a second aspect of an embodiment of the present application, a three-dimensional reconstruction device is provided, which includes: a determination module, which is used to determine the characteristics of a preset object based on initial scanning data of the preset object; the determination module is also used to determine the target scanning mode corresponding to different areas of the preset object based on the characteristics; an acquisition module is used to respond to the target scanning mode selected by a user, start the selected target scanning mode to scan the corresponding area of ​​the preset object, and obtain the scanning data of each area of ​​the preset object; and a reconstruction module is used to fuse the scanning data of all areas of the preset object to obtain a three-dimensional reconstructed model of the preset object.

[0014] A third aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the three-dimensional reconstruction method.

[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional reconstruction method is implemented.

[0016] The 3D reconstruction method provided in the embodiment of the present application automatically determines the target scanning mode for different areas of the preset object by intelligently analyzing the characteristics of the preset object, starts the corresponding scanning mode in response to the user's selection to obtain the scanning data of each area, and finally fuses all the data to construct a high-precision 3D reconstruction model. It can take into account both scanning efficiency and detail retention, improve the integrity of the scanning data, and thus improve the accuracy of the 3D reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of an application environment of a three-dimensional reconstruction method provided in one embodiment of the present application.

[0019] Figure 2 A schematic diagram of a flow chart of a three-dimensional reconstruction method provided in one embodiment of the present application.

[0020] Figure 3 A flowchart of specific implementation steps of step S204 provided in an embodiment of the present application.

[0021] Figure 4 A structural diagram of a three-dimensional reconstruction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specified order or sequence.

[0024] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0025] In the field of three-dimensional scanning technology, fixed scanning scenes are particularly widely used, for example, including but not limited to industrial design scenes, cultural relics protection scenes, oral medicine scenes, etc. However, traditional fixed scanning technology often has an irreconcilable contradiction between the pursuit of scanning details and scanning range. On the one hand, in order to ensure that the details of the scanned object are clear enough, the scanning range is often limited to a smaller area, so that a complete scan of an object requires multiple scans, which takes a lot of time. On the other hand, although there are monocular and binocular combined scanning solutions to try to solve this problem, the limitations of the scanning range still exist, and it is impossible to retain sufficiently good object details while improving scanning efficiency.

[0026] In one example, when using an ordinary camera for long-distance shooting, although it can cover a larger scanning range, the details in the scanned data often become blurred due to the long distance, and cannot meet the needs of high-precision three-dimensional reconstruction. Secondly, when scanning objects with complex structures, such as grooves or deep holes, it is necessary to scan each small area closely, which is not only time-consuming and labor-intensive, but also the errors of multiple scans will be superimposed like the misalignment of puzzle pieces, eventually leading to deformation and distortion of the three-dimensional reconstructed model. In addition, when dealing with such complex structures, existing three-dimensional scanning technologies mostly use manual adjustment of the scanning position, which not only increases the complexity of the operation, but also further reduces the scanning efficiency.

[0027] In order to solve the above problems, the three-dimensional reconstruction method provided in the embodiment of the present application automatically determines the target scanning mode for different areas of the preset object by intelligently analyzing the characteristics of the preset object, starts the corresponding scanning mode in response to the user's selection to obtain the scanning data of each area, and finally fuses all the data to construct a high-precision three-dimensional reconstruction model. It can take into account both scanning efficiency and detail retention, improve the integrity of the scanning data, and thus improve the accuracy of the three-dimensional reconstruction results.

[0028] See also Figure 1 , is a schematic diagram of an application environment of a three-dimensional reconstruction method provided by an embodiment of the present application. Figure 1 As shown, the three-dimensional reconstruction method provided in the embodiment of the present application can be applied to an electronic device 10, and the electronic device 10 can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, an energy storage device, a power distribution device, a vehicle-mounted device, a self-moving device, a scanner, and other electronic devices. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0029] like Figure 1As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the I / O interface 104 through the bus 105.

[0030] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), Bluetooth (blue tooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0031] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0032] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 103. The non-volatile memory may include a disk storage device and a flash memory.

[0033] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions, and when the multiple instructions are executed by the processor 103, the three-dimensional reconstruction method executed on the electronic device 10 can be implemented.

[0034] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0035] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0036] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned three-dimensional reconstruction method.

[0037] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information or visualize information. In addition, the I / O interface 104 can also be used to connect various sensor devices, such as an image sensor, etc., to obtain required image data and other data.

[0038] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0039] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0040] In one example, the electronic device 10 may be a scanner, for example, a scanner including at least two image sensors. Any one of the at least two image sensors may be used to perform a monocular scanning operation, and any two of the at least two image sensors may be used to perform a binocular scanning operation, which is not specifically limited in the present application.

[0041] The following will be a computer program product that implements the three-dimensional reconstruction method of the embodiment of the present application and runs on an electronic device (such as Figure 1 The electronic device 10 shown in FIG. Figure 2 FIG. 1 is a flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application. In an embodiment of the present application, the method includes the following steps: Step S201, determining the features of a preset object according to the scan data of the preset object.

[0042] In some embodiments, the preset object may include any object that needs to be reconstructed in three dimensions, such as a patient's oral cavity. Before determining the characteristics of the preset object according to the scan data of the preset object, the method further includes: in response to the user's operation of starting scanning, scanning the preset object using an initial scanning mode to obtain the initial scanning data. The initial scanning mode includes but is not limited to a monocular scanning mode and a binocular scanning mode, and can be set based on system default settings or user preferences.

[0043] In one example, in order to facilitate user operation, a button for starting scanning can be set in the graphical interactive interface, and scanning is started in response to the user's selection of the button, and the preset object is scanned using the default initial scanning mode to obtain initial scanning data. Among them, the scanning range corresponding to the initial scanning mode can be a preset range, for example, the preset range can represent the range when scanning at a longer distance. In another example, the initial scanning mode can also be determined based on the scanning mode selected by the user, and this application does not specifically limit this.

[0044] If the initial scanning mode is a monocular scanning mode, the electronic device uses a single image sensor to obtain image data of the preset object. For example, a camera is used to shoot the preset object from a fixed position of the preset object to obtain two-dimensional global image data of the preset object. The monocular scanning mode can quickly obtain the overall shape and outline of the preset object, providing a basis for subsequent feature recognition and analysis.

[0045] If the initial scanning mode is a binocular scanning mode, the electronic device uses two image sensors to simulate the stereoscopic vision principle of the human eye to scan the preset object. For example, two cameras simultaneously shoot the preset object from different angles, and obtain the depth information of the preset object by calculating the parallax between the two images. The initial scanning data generated by the binocular scanning mode not only contains the two-dimensional image data of the preset object, but also includes the point cloud data or depth map with three-dimensional spatial information generated by the binocular algorithm, which can provide more detailed and accurate basic data for subsequent three-dimensional reconstruction.

[0046] Based on the above embodiment, the initial scanning mode can be started according to the user operation to quickly scan the preset object, and the initial scanning data of the object can be quickly obtained, laying the foundation for subsequent feature analysis and scanning mode selection.

[0047] In some embodiments, the initial scanning data includes image data, and determining the features of the preset object based on the initial scanning data of the preset object includes: performing semantic segmentation on the preset object based on the image data, and determining multiple different areas of the preset object based on the results of the semantic segmentation; determining the feature point density and depth change rate of each area of ​​the preset object, and determining the features based on the feature point density and the depth change rate.

[0048] In some embodiments, when the preset object is semantically segmented according to the image data, each pixel or region in the image can be classified into a specific semantic category, such as teeth, gums, holes caused by tooth decay, etc., so that the preset object is divided into multiple different regions according to the semantic category. In one example, the geometric area of ​​each region can be calculated, for example, the number of pixels contained in each region. In another example, a semantic category label can be set for each region based on the semantic category obtained by semantic segmentation, for example, the semantic category label of a certain region can be a hole.

[0049] In some embodiments, for each area identified by semantic segmentation, its feature point density can be further calculated, such as the number of feature points per unit area. These feature points can be corner points, edge points, or points with significant texture changes in the image. The density of feature points can intuitively reflect the richness of the texture features of the area. The depth change rate of each area can also be calculated, for example, by comparing the depth values ​​of adjacent pixels or areas to calculate the depth change rate. The depth change rate can reflect the concavity or structural complexity of the surface of each area. For example, the depth change rate of a hole structure or an uneven surface will be greater than the depth change rate of a plane.

[0050] In some embodiments, the features of each region of the preset object can be determined based on the feature point density and depth change rate of each region of the preset object. For example, if the feature point density of a region is high and the depth change rate is also large, then the region may have rich texture features and complex structures; on the contrary, if the feature point density of a region is low and the depth change rate is also small, then the region may be relatively smooth and lack significant texture or structural features.

[0051] In other embodiments, the feature of each region may be determined based on the geometric area and semantic category label of each region of the preset object. There may be an association between the semantic category label and the depth change rate, for example, the depth change rate of the hole category may be greater than a preset change rate threshold.

[0052] In this way, multiple features about the preset object can be extracted from the initial scanning data, and these features have important reference value for subsequent tasks such as scanning pattern determination or reconstruction.

[0053] Step S202: determining target scanning modes corresponding to different areas of the preset object according to the features.

[0054] In some embodiments, different areas of the preset object can be divided into multiple types of areas according to the features, and a corresponding target scanning mode is assigned to each type of area. In one example, the multiple types of areas include but are not limited to: first type of areas, second type of areas, third type of areas and fourth type of areas. The target scanning mode includes at least one of the following: a binocular scanning mode in a first range, a monocular and binocular combination mode in a first range, a binocular scanning mode in a second range, and a monocular and binocular combination mode in a second range. Among them, the first range is larger than the second range, and the monocular and binocular combination mode represents a combination mode of a monocular scanning mode and a binocular scanning mode.

[0055] In some embodiments, the first type of area represents an area whose geometric area is greater than a preset area threshold, whose feature point density is less than or equal to a preset density threshold, and whose depth change rate is less than or equal to a preset change rate threshold. Among them, the preset density threshold, the preset density threshold and the preset change rate threshold can be set according to actual needs, and the present application does not impose specific restrictions on this. The first type of area is usually relatively flat and has a simple texture, which is suitable for using a large range of binocular scanning mode. Therefore, it can be determined that the target scanning mode corresponding to the first type of area is a binocular scanning mode of the first range. Among them, the first range can represent a larger range, and the first range can be smaller than the preset range described in step S201. In this way, the binocular scanning mode of the first range can quickly capture the overall structure of the first type of area while ensuring the continuity of the scan and the integrity of the data. Since the first type of area is usually relatively flat, the binocular scanning mode can accurately calculate the depth information without relying too much on the supplement of monocular scanning.

[0056] In some embodiments, the second type of area represents an area whose geometric area is less than or equal to a preset area threshold, whose feature point density is greater than a preset density threshold, and whose depth change rate is less than or equal to a preset change rate threshold. Although such areas may be small in area, they are rich in texture and are suitable for using a small-range binocular scanning mode to capture more details. Therefore, it can be determined that the target scanning mode corresponding to the second type of area is a binocular scanning mode of the second range. Among them, the second range can represent a smaller range, and the second range can be smaller than the first range. In this way, in the binocular scanning mode of the second range, the scanning range can be precisely controlled, and can be focused on the details of the second type of area, ensuring accurate capture of textures and microstructures during the scanning process. At the same time, scanning in a small range also helps to reduce unnecessary calculations and improve scanning efficiency.

[0057] In some embodiments, the third category of areas represents areas whose geometric area is greater than a preset area threshold and whose depth change rate is greater than a preset change rate threshold. Such areas are large in area and complex in structure, and a wide range of combined monocular and binocular scanning modes can be used to obtain complete scanning data. Therefore, it can be determined that the target scanning mode corresponding to the third category of areas is the monocular and binocular combination mode of the first range. In this way, the monocular scanning mode in the monocular and binocular combination mode of the first range can cover a wider range of third category areas, while the binocular scanning mode can provide accurate depth information in the key areas of the third category of areas. This combination mode not only improves the integrity of the data, but also ensures the accurate reconstruction of areas with complex structures.

[0058] In some embodiments, the fourth type of area represents an area whose geometric area is less than or equal to a preset area threshold and whose depth change rate is greater than a preset change rate threshold. Such areas are usually small in area and extremely complex in structure (such as deep holes or small grooves). A small range of combined monocular and binocular scanning modes can be used to improve data integrity and accuracy. Therefore, it can be determined that the target scanning mode corresponding to the fourth type of area is the monocular and binocular combination mode of the second range. In this way, in the monocular and binocular combination mode of the second range, the scanning range is further narrowed, which can deeply explore the fine structure of the fourth type of area, while combining the advantages of monocular and binocular scanning to ensure the accuracy and integrity of the data.

[0059] In other embodiments, the category of each area can also be determined according to the semantic category of each area, and a corresponding target scanning mode can be assigned to it. For example, an area with a semantic category of hole can be determined as a fourth type of area, and a second range of monocular and binocular combination mode can be assigned to it.

[0060] In some other embodiments, the multiple categories of areas may also include a fifth category of areas. For example, the fifth category of areas represents an area whose geometric area is greater than a preset area threshold, the feature point density is greater than a preset density threshold, and the depth change rate is less than or equal to a preset change rate threshold. For the fifth category of areas, due to their large area and complex texture, a scanning mode that can cover a wide area and capture detail information at the same time can be adopted. For example, it can be determined that the target scanning mode corresponding to the fifth category of areas is a binocular scanning mode in the first range to ensure that while capturing the overall structure, the texture details can also be accurately recorded. If the texture is extremely complex or requires higher precision, it can also be considered to combine the monocular scanning mode for local supplementary scanning, for example, using the monocular and binocular combination mode in the first range to further improve data integrity and accuracy.

[0061] In some further embodiments, the multiple categories of areas may also include a sixth category of areas, for example, the sixth category of areas represents an area whose geometric area is less than or equal to a preset area threshold, whose feature point density is less than or equal to a preset density threshold, and whose depth change rate is greater than a preset change rate threshold. For the sixth category of areas, since their area is small but the depth changes dramatically, a small-range but high-precision binocular scanning mode can be adopted, combined with a monocular scanning mode for local depth measurement. For example, it can be determined that the target scanning mode corresponding to the sixth category of areas is a monocular and binocular combination mode of the second range to ensure accurate recording of depth changes in a limited space.

[0062] Based on the above embodiments, different areas of the preset object can be carefully divided based on the characteristics of each area, and the corresponding target scanning mode can be intelligently allocated according to the characteristics of each area, such as the geometric area, feature point density, and depth change rate, which can significantly improve the efficiency and accuracy of three-dimensional reconstruction.

[0063] In some embodiments, different identifiers may be used to mark different areas in the interactive interface. For example, different identifiers may be different numbers, colors, symbols, and the like.

[0064] Step S203, in response to the target scanning mode selected by the user, starting the selected target scanning mode to scan the corresponding area of ​​the preset object, and obtaining scanning data of each area of ​​the preset object.

[0065] In some embodiments, all optional target scanning modes may be displayed through a preset interactive interface, and a brief description or icon may be attached next to each target scanning mode to help the user understand the characteristics and applicable scenarios of each mode. In response to the user selecting a target scanning mode desired to be used for scanning a specific area of ​​a preset object through the interactive interface, the selected target scanning mode is started. For example, in response to the user selecting an icon of a target scanning mode, the selected target scanning mode is started.

[0066] In some embodiments, the electronic device starts the target scanning mode selected by the user and configures the corresponding scanning parameters, such as scanning speed, resolution, focal length, scanning range, etc. According to the characteristics of the selected target scanning mode, the electronic device automatically adjusts the settings of the scanning device to ensure the best scanning effect. After starting the scanning mode, the electronic device uses the corresponding scanning technology to scan the specified area of ​​the preset object. During the scanning process, the electronic device can display the scanning progress, the visual preview of the scanning area, or the preliminary processing results of the scanning data in real time through the interactive interface. After the scanning is completed, the electronic device collects and stores the scanning data obtained from each area of ​​the preset object. The scanning data may include point cloud data, texture information, depth information, etc., depending on the type of the selected scanning mode and the performance of the scanning device. The obtained scanning data can be optimized. For example, the optimization process may include but is not limited to denoising, filtering, etc. to improve the data quality of the scanning data. In this way, a high-quality scanning data set corresponding to each area can be obtained.

[0067] Based on the above embodiments, various areas of the preset object can be scanned in the most suitable manner according to the user's selection, solving the contradiction between the pursuit of scanning details and the scanning range in a fixed scanning scene, avoiding the limitation of the scanning range, and retaining sufficiently good object details while improving the scanning efficiency, so that high-quality scanning data can be obtained efficiently, providing a solid foundation for subsequent tasks such as three-dimensional reconstruction.

[0068] Step S204: fusing the scanned data of all regions of the preset object to obtain a three-dimensional reconstructed model of the preset object.

[0069] In some embodiments, when performing three-dimensional reconstruction on the scan data of all areas of a preset object, if the scan data includes monocular scan data and binocular scan data, the coordinates of the monocular and binocular scan data can be aligned, and then redundant information of the overlapping area can be identified and removed, and then global fusion is performed based on the data after removing the overlapping area, and finally the fused data can be deeply processed using a global optimization algorithm, so as to accurately construct a three-dimensional model of the preset object. In one example, the specific implementation steps of step S204 include the following: Figure 3 The process shown.

[0070] The 3D reconstruction method provided in the embodiment of the present application can accurately identify the features of the preset object, intelligently select the scanning mode that best suits each area, and scan each area of ​​the preset object in a targeted manner, and finally seamlessly merge the scanning data of all areas to generate an accurate and complete 3D reconstruction model. This can not only improve the efficiency and accuracy of 3D reconstruction, but also ensure that the model can truly reflect the complex structure and detailed features of the preset object.

[0071] In some embodiments, reference Figure 3 As shown, the specific implementation steps of step S204 include the following process: Step S301: if the scan data includes monocular scan data and binocular scan data, coordinate alignment is performed on the monocular scan data and the binocular scan data.

[0072] In some embodiments, if the scan data includes monocular scan data and binocular scan data, the two types of data can be aligned to convert the data acquired in different scanning modes into a unified coordinate system, effectively ensuring the spatial consistency of the data. In one example, the algorithm used for coordinate alignment can include but is not limited to feature point matching, iterative closest point (ICP) algorithm or other related algorithms.

[0073] In some embodiments, coordinate alignment can not only convert monocular scanning data and binocular scanning data into a unified coordinate system, but also integrate scanning data obtained in different areas due to differences in scanning ranges into this unified coordinate system. In this way, it can be ensured that data from different scanning sources, different scanning positions and even different scanning time points can be compared and analyzed in the same spatial framework, so that subsequent 3D reconstruction can be based on a coherent and unambiguous data set to build a more accurate 3D model.

[0074] Step S302, determining an overlapping area between the monocular scanning data and the binocular scanning data after coordinate alignment, and removing redundant data in the overlapping area.

[0075] In some embodiments, after the coordinate alignment is completed, the overlapping areas of the monocular scanning data and the binocular scanning data can be identified in a unified coordinate system, and the redundant data of these overlapping areas can be removed. In one example, based on the spatial geometric relationship or image matching algorithm between the monocular scanning data and the binocular scanning data, the data points in the monocular scanning data and the binocular scanning data that are less than a preset distance threshold in spatial distance can be determined, and such data points can be used as redundant data in the overlapping area. When removing redundant data, data points with higher accuracy and richer information, such as points in binocular scanning data, can be preferentially retained. In this way, it can help reduce data redundancy and improve the accuracy and efficiency of three-dimensional reconstruction.

[0076] Step S303 , performing global data fusion based on the scan data obtained after removing redundant data from all regions to obtain global fused data.

[0077] In some embodiments, global data fusion refers to effectively integrating information from multiple data sources to form a more comprehensive and accurate data set. In the field of 3D reconstruction, global data fusion usually includes splicing and fusing point clouds, meshes or voxels from multiple scan data sets to form a complete 3D model.

[0078] In some embodiments, when the global data is fused, the scan data after removing the redundant data of all regions can be globally spliced ​​according to the positions of the preset marker points of the preset object to obtain the global fused data. In one example, the marker points can be points with clear geometric features, such as corner points, edge points, etc., or they can be specific objects or structures determined by semantic recognition, such as the corner points of the teeth in the patient's mouth, the edge points of the gums, etc. By determining the positions of these marker points in each scan data set, accurate alignment and splicing between the scan data can be achieved.

[0079] In one example, landmark points can be extracted from each scan data set by an image processing algorithm or a machine learning algorithm. For example, the image processing algorithm may include but is not limited to corner detection, edge detection and other algorithms, and the machine learning algorithm may include but is not limited to feature matching, deep learning and other algorithms. Then, by calculating the spatial distance and angle relationship between the landmark points, these landmark points are used to match the scan data sets to ensure accurate alignment between the scan data sets. After the landmark points are matched, the transformation matrix of the same landmark point in different scan data sets, such as the rotation matrix and translation matrix, can be calculated, and the scan data sets can be spliced ​​using the transformation matrix to ensure seamless connection between scan data in different regions and obtain globally fused data.

[0080] Step S304: globally optimize the global fusion data to obtain the three-dimensional reconstructed model.

[0081] In some embodiments, the global fusion data may still contain some errors, inconsistencies or overlapping areas, and the global fusion data may be further globally optimized and processed to eliminate these potential problems and ultimately generate a high-quality three-dimensional reconstructed model.

[0082] In some embodiments, there may still be overlapping areas between the scanned data of the global fusion data, for example, overlapping areas caused by the edges of the scanned data sets of different regions after splicing, and these overlapping areas may cause redundancy or errors in the three-dimensional model. Therefore, the global fusion data can be de-overlapped, and the binocular scan data can be retained in the de-overlapping operation. In this way, in the global fusion data after the de-overlapping operation, there is no overlapping area between the edges of different regions, which can improve the accuracy of the three-dimensional model.

[0083] In some embodiments, after the de-overlapping operation, there may be problems of discontinuous edges or unnatural transitions between the scanned data of different regions. A uniform point spacing transition algorithm may be used to perform smooth transition processing on the edges of the scanned data of different regions. The uniform point spacing transition algorithm may determine the parameters and algorithms required for smooth transition by calculating the point cloud density and geometric features between the scanned data of different regions. In this way, the edge continuity and natural transition between the scanned data of different regions may be improved, thereby improving the accuracy of the three-dimensional model.

[0084] In some embodiments, after performing the above optimization, the global fusion data may be error corrected and optimized as a whole. For example, the global fusion data may be error corrected and optimized as a whole by comparing the spatial position relationship, geometric features, etc. between different scan data. Among them, error correction methods such as the least squares method and the iterative closest point algorithm may be used to eliminate errors and inconsistencies between scan data. In addition, overall optimization methods such as mesh optimization and surface reconstruction may be used to improve the accuracy and integrity of the three-dimensional reconstructed model.

[0085] Based on the above embodiments, the data of different scanning modes and regions can be integrated into a unified coordinate system through coordinate alignment of monocular and binocular scanning data, removal of redundant data in overlapping areas, global data fusion, and subsequent global optimization processing, and data redundancy, errors, and inconsistencies can be effectively eliminated. In this way, it can be ensured that the 3D reconstruction model can be constructed based on a coherent and unambiguous data set, thereby significantly improving the accuracy, integrity, and natural transition of the 3D model, and providing high-quality 3D visualization solutions for various application scenarios.

[0086] Figure 4 It is a structural diagram of a three-dimensional reconstruction device provided in one embodiment of the present application.

[0087] In some embodiments, the 3D reconstruction device 40 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the 3D reconstruction device 40 may be stored in a memory of an electronic device and executed by at least one processor to perform (see Figure 2 Description) Function of three-dimensional reconstruction.

[0088] In this embodiment, the three-dimensional reconstruction device 40 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: a determination module 401, an acquisition module 402, and a reconstruction module 403. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functional implementation of each module in the three-dimensional reconstruction device 40 can refer to the above definition of the three-dimensional reconstruction method, and will not be repeated here.

[0089] The determination module 401 is used to determine the characteristics of the preset object according to the initial scanning data of the preset object.

[0090] The determination module 401 is further configured to determine target scanning modes corresponding to different regions of the preset object according to the features.

[0091] The acquisition module 402 is used to respond to the target scanning mode selected by the user, start the selected target scanning mode to scan the corresponding area of ​​the preset object, and obtain the scanning data of each area of ​​the preset object.

[0092] The reconstruction module 403 is used to fuse the scan data of all regions of the preset object to obtain a three-dimensional reconstructed model of the preset object.

[0093] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0094] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0095] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0096] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specified application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specified application, but such implementation should not be considered to be beyond the scope of this application.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Determining the characteristics of the preset object according to the initial scan data of the preset object; Determining target scanning modes corresponding to different areas of the preset object according to the features; In response to a target scanning mode selected by a user, starting the selected target scanning mode to scan a corresponding area of ​​the preset object, and obtaining scanning data of each area of ​​the preset object; The scan data of all regions of the preset object are fused to obtain a three-dimensional reconstructed model of the preset object.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: Before determining the features of the preset object according to the scanned data of the preset object, the method further includes: In response to the scanning start operation performed by the user, the preset object is scanned using an initial scanning mode to obtain the initial scanning data, wherein the initial scanning mode includes a monocular scanning mode and a binocular scanning mode.

3. The three-dimensional reconstruction method according to claim 1, characterized in that: The initial scan data includes image data, and determining the features of the preset object according to the initial scan data of the preset object includes: Performing semantic segmentation on the preset object according to the image data, and determining a plurality of different regions of the preset object according to a result of the semantic segmentation; Determine the density of feature points and the depth change rate of each area of ​​the preset object, and determine the feature according to the density of feature points and the depth change rate.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The target scanning mode includes at least one of the following: a binocular scanning mode in a first range, a monocular and binocular combination mode in a first range, a binocular scanning mode in a second range, and a monocular and binocular combination mode in a second range, wherein the first range is larger than the second range, and the monocular and binocular combination mode represents a combination mode of a monocular scanning mode and a binocular scanning mode.

5. The three-dimensional reconstruction method according to claim 4, characterized in that: Determining the target scanning modes corresponding to different areas of the preset object according to the features includes: Dividing different regions of the preset object into multiple regions according to the features, the multiple regions including a first region, a second region, a third region and a fourth region; Determine that the target scanning mode corresponding to the first type of area is a binocular scanning mode of the first range; Determine that the target scanning mode corresponding to the second type of area is a binocular scanning mode of the second range; Determine that the target scanning mode corresponding to the third type of area is a monocular and binocular combination mode of the first range; Determine that the target scanning mode corresponding to the fourth type of area is a monocular and binocular combination mode of the second range.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The first type of region represents a region whose geometric area is greater than a preset area threshold, whose feature point density is less than or equal to a preset density threshold, and whose depth change rate is less than or equal to a preset change rate threshold; The second type of region represents a region whose geometric area is less than or equal to the preset area threshold, whose feature point density is greater than the preset density threshold, and whose depth change rate is less than or equal to the preset change rate threshold; The third type of area represents an area whose geometric area is greater than the preset area threshold and whose depth change rate is greater than the preset change rate threshold; The fourth type of region represents a region whose geometric area is less than or equal to the preset area threshold and whose depth change rate is greater than the preset change rate threshold.

7. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of fusing the scanned data of all regions of the preset object to obtain a three-dimensional reconstructed model of the preset object includes: If the scanning data includes monocular scanning data and binocular scanning data, coordinate alignment is performed on the monocular scanning data and the binocular scanning data; Determine an overlapping area between the monocular scanning data and the binocular scanning data after coordinate alignment, and remove redundant data in the overlapping area; Based on the scan data obtained after removing redundant data from all regions, global data fusion is performed to obtain global fused data; The global fusion data is globally optimized to obtain the three-dimensional reconstructed model.

8. The three-dimensional reconstruction method according to claim 7, characterized in that: The scanning data obtained after removing redundant data from all regions is subjected to global data fusion to obtain global fused data, including: According to the positions of the preset marking points of the preset objects, the scan data after the redundant data of all regions are removed are globally spliced ​​to obtain the global fusion data.

9. The three-dimensional reconstruction method according to claim 7, characterized in that: The optimizing of the global fusion data comprises: Performing a de-overlapping operation on the global fusion data, and retaining the binocular scanning data in the de-overlapping operation; The uniform point distance transition algorithm is used to perform smooth transition processing on the edges of scanned data in different areas.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory storing one or more application programs, wherein the one or more application programs are configured to implement the three-dimensional reconstruction method according to any one of claims 1 to 9 when executed by the processor.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of an electronic device, the three-dimensional reconstruction method according to any one of claims 1 to 9 is implemented.

Images