3D Reconstruction Method, Electronic Device, and Storage Medium

Abstract

An embodiment of the present application provides a three-dimensional reconstruction method, an electronic device, and a storage medium. The method includes: determining the features of a preset object according to the initial scan data of the preset object; determining a target scan mode corresponding to different regions of the preset object according to the features; in response to the target scan mode selected by the user, starting the selected target scan mode to scan the corresponding region of the preset object, and obtaining the scan data of each region of the preset object; fusing the scan data of all regions of the preset object to obtain a three-dimensional reconstruction model of the preset object. Using the above method can improve the efficiency and accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology. Specifically, it relates to a three-dimensional reconstruction method, an electronic device, and a storage medium. Background Art

[0002] Existing three-dimensional scanning technologies face great limitations in the scanning of fixed scenarios, resulting in poor effects of the three-dimensional reconstruction model. For example, when using an ordinary camera, in order to see the entire object clearly, it can only be photographed from a long distance, resulting in blurred details in the scanned data. In addition, when scanning an object area with a complex structure (such as a groove), it is necessary to scan each small area closely and repeatedly adjust the position to complete the stitching. This not only takes time, but also the errors in multiple scans will be superimposed like a jigsaw puzzle misalignment, resulting in deformation of the final model. Summary of the Invention

[0003] This application discloses a three-dimensional reconstruction method, an electronic device, and a storage medium, which can switch the scanning mode according to the regional characteristics of the scanned object, improve the scanning efficiency while retaining sufficient scanning details, and solve the contradiction between scanning efficiency and detail retention in a fixed scanning scenario.

[0004] In a first aspect of an embodiment of this application, a three-dimensional reconstruction method is provided, including: determining the characteristics of a preset object according to the initial scan data of the preset object; determining the target scan mode corresponding to different regions of the preset object according to the characteristics; in response to the target scan mode selected by the user, starting the selected target scan mode to scan the corresponding region of the preset object, and obtaining the scan data of each region of the preset object; fusing the scan data of all regions of the preset object to obtain a three-dimensional reconstruction model of the preset object.

[0005] According to an embodiment of this application, before determining the characteristics of the preset object according to the scan data of the preset object, the method further includes: in response to the operation of the user to start scanning, scanning the preset object in an initial scan mode to obtain the initial scan data, where the initial scan mode includes a monocular scan mode and a binocular scan mode.

[0006] According to an embodiment of this application, the initial scan data includes image data, and determining the characteristics of the preset object according to the initial scan data of the preset object includes: performing semantic segmentation on the preset object according to the image data, and determining multiple different regions of the preset object according to the result of the semantic segmentation; determining the feature point density and depth change rate of each region of the preset object, and determining the characteristics according to the feature point density and the depth change rate.

[0007] According to an embodiment of the present application, the target scanning mode includes at least one of the following: a binocular scanning mode in a first range, a combined single-binocular mode in the first range, a binocular scanning mode in a second range, and a combined single-binocular mode in the second range, where the first range is greater than the second range, and the combined single-binocular mode represents a combined mode of a monocular scanning mode and a binocular scanning mode.

[0008] According to an embodiment of the present application, the determining the target scanning mode corresponding to different regions of the preset object according to the feature includes: dividing different regions of the preset object into multiple types of regions according to the feature, where the multiple types of regions include a first type of region, a second type of region, a third type of region, and a fourth type of region; determining the target scanning mode corresponding to the first type of region as the binocular scanning mode in the first range; determining the target scanning mode corresponding to the second type of region as the binocular scanning mode in the second range; determining the target scanning mode corresponding to the third type of region as the combined single-binocular mode in the first range; and determining the target scanning mode corresponding to the fourth type of region as the combined single-binocular mode in the second range.

[0009] According to an embodiment of the present application, the first type of region represents a region where the geometric area is greater than a preset area threshold, the feature point density is less than or equal to a preset density threshold, and the depth change rate is less than or equal to a preset change rate threshold; the second type of region represents a region where the geometric area is less than or equal to the preset area threshold, the feature point density is greater than the preset density threshold, and the depth change rate is less than or equal to the preset change rate threshold; the third type of region represents a region where the geometric area is greater than the preset area threshold and the depth change rate is greater than the preset change rate threshold; and the fourth type of region represents a region where the geometric area is less than or equal to the preset area threshold and the depth change rate is greater than the preset change rate threshold.

[0010] According to an embodiment of the present application, the fusing the scanning data of all regions of the preset object to obtain a three-dimensional reconstruction model of the preset object includes: if the scanning data includes monocular scanning data and binocular scanning data, aligning the coordinates of the monocular scanning data and the binocular scanning data; determining an overlapping region of the monocular scanning data and the binocular scanning data after coordinate alignment, and removing redundant data in the overlapping region; performing global data fusion based on the scanning data obtained after removing redundant data from all regions to obtain global fusion data; and performing global optimization on the global fusion data to obtain the three-dimensional reconstruction model.

[0011] According to an embodiment of the present application, global data fusion is performed on the scanned data obtained after removing redundant data from all regions to obtain global fusion data, including: globally stitching the scanned data after removing redundant data from all regions according to the positions of preset landmark points of the preset object to obtain the global fusion data.

[0012] According to an embodiment of the present application, the optimization of the global fusion data includes: performing an overlapping removal operation on the global fusion data and retaining binocular scanned data during the overlapping removal operation; using a uniform point distance transition algorithm to perform smooth transition processing on the edges of scanned data in different regions.

[0013] A three-dimensional reconstruction device is provided in a second aspect of an embodiment of the present application. The three-dimensional reconstruction device includes: a determination module for determining the features of the preset object according to the initial scanned data of the preset object; the determination module is further configured to determine a target scanning mode corresponding to different regions of the preset object according to the features; an acquisition module for starting the selected target scanning mode to scan the corresponding region of the preset object in response to the target scanning mode selected by the user, and acquiring the scanned data of each region of the preset object; a reconstruction module for fusing the scanned data of all regions of the preset object to obtain a three-dimensional reconstruction model of the preset object.

[0014] An electronic device is provided in a third aspect of an embodiment of the present application, including: one or more processors; a memory; and one or more application programs, where the one or more application programs are stored in the memory and configured to be executed by the processor to implement the three-dimensional reconstruction method.

[0015] A computer-readable storage medium is provided in a fourth aspect of an embodiment of the present application. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the three-dimensional reconstruction method is implemented.

[0016] The three-dimensional reconstruction method provided by the embodiment of the present application can automatically determine the target scanning modes of different regions of the preset object by intelligently analyzing the features of the preset object, start the corresponding scanning mode to obtain the scanned data of each region after the user's selection, and finally fuse all the data to construct a high-precision three-dimensional reconstruction model, which can balance the scanning efficiency and detail retention, improve the integrity of the scanned data, and thus improve the accuracy of the three-dimensional reconstruction result. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of the application environment of a three-dimensional reconstruction method provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic flowchart of the three-dimensional reconstruction method provided in an embodiment of the present application.

[0020] Figure 3 It is a schematic flowchart of the specific implementation steps of step S204 provided in an embodiment of the present application.

[0021] Figure 4 It is a structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application. Detailed implementation manners

[0022] To make the objectives, technical solutions, and advantages of the present application clearer, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] It should be noted that in the present application, "at least one" means one or more, and "a plurality" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and drawings of the present application are used to distinguish similar objects, rather than to describe a specified order or sequence.

[0024] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly, using words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0025] In the field of three-dimensional scanning technology, the application of fixed scanning scenarios is particularly extensive. For example, it includes, but is not limited to, industrial design scenarios, cultural relic protection scenarios, oral medicine scenarios, etc. However, there are often irreconcilable contradictions between the pursuit of scanning details and the scanning range in traditional fixed scanning technologies. On the one hand, in order to ensure that the details of the scanned object are clear enough, the scanning range is often limited to a small area, so that multiple scans are required to completely scan an object, consuming a large amount of time. On the other hand, although there are already single binocular combined scanning solutions trying to solve this problem, the limitation of the scanning range still exists, and it is impossible to retain good enough object details while improving the scanning efficiency.

[0026] In one example, when using an ordinary camera for long-distance shooting, although a large scanning range can be covered, the details in the scanning data often become blurred due to the excessive distance, which cannot meet the requirements of high-precision three-dimensional reconstruction. Secondly, when scanning an object with a complex structure, such as areas with grooves or deep holes, it is necessary to scan each small area closely, which is not only time-consuming and laborious, but also the errors of multiple scans will be superimposed like a jigsaw puzzle misalignment, ultimately resulting in the deformation and distortion of the three-dimensional reconstruction model. In addition, when dealing with such complex structures, existing three-dimensional scanning technologies mostly adopt the method of manually adjusting the scanning position, which not only increases the complexity of the operation, but also further reduces the scanning efficiency.

[0027] To solve the above problems, the three-dimensional reconstruction method provided in the embodiments of the present application can automatically determine the target scanning modes of different regions of a preset object by intelligently analyzing the characteristics of the preset object, start the corresponding scanning mode to obtain the scanning data of each region after the user's selection, and finally fuse all the data to construct a high-precision three-dimensional reconstruction model. It can balance the scanning efficiency and detail retention, improve the integrity of the scanning data, and thus improve the accuracy of the three-dimensional reconstruction result.

[0028] Please refer to Figure 1 , which is a schematic diagram of the application environment of a three-dimensional reconstruction method provided in an embodiment of the present application. As Figure 1 shown, the three-dimensional reconstruction method provided in the embodiments of the present application can be applied to an electronic device 10. The electronic device 10 can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, a netbook, an energy storage device, a power distribution device, a vehicle-mounted device, a self-mobile device, a scanner, and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device.

[0029] As Figure 1As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is respectively coupled to the communication module 101, the memory 102, and the I / O interface 104 through the bus 105.

[0030] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more of the solutions for wired communication such as a universal serial bus (USB), a controller area network bus (CAN). The wireless communication module may provide one or more of the solutions for wireless communication such as wireless fidelity (Wi-Fi), Bluetooth (BT), a mobile communication network, frequency modulation (FM), near field communication (NFC), and infrared technology (IR).

[0031] The memory 102 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The random access memory can be directly read and written by the processor 103 and can be used to store an operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data, etc. The random access memory may include a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0032] The non-volatile memory can also store executable programs and store user and application data, etc., and can be pre-loaded into the random access memory for the processor 103 to directly read and write. The non-volatile memory may include a disk storage device and a flash memory.

[0033] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 103, a three-dimensional reconstruction method executable on the electronic device 10 can be implemented.

[0034] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to implement the expansion of the storage capacity of the electronic device 10.

[0035] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0036] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute the computer program stored in the memory 102 to implement the above three-dimensional reconstruction method.

[0037] The I / O interface 104 is used to provide channels for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information or visualize information. In addition, the I / O interface 104 can also be used to connect various sensor devices, such as an image sensor, etc., so as to obtain data such as required image data.

[0038] The bus 105 is at least used to provide a communication channel for mutual communication between the communication module 101, the memory 102, the processor 103, and the I / O interface 104 in the electronic device 10.

[0039] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0040] In one example, the electronic device 10 may be a scanner, for example, a scanner including at least two image sensors. Any one of the at least two image sensors can be used to perform a monocular scanning operation, and any two of the at least two image sensors can be used to perform a binocular scanning operation. The present application does not make specific limitations on this.

[0041] The following will take the computer program product implementing the 3D reconstruction method of the embodiments of the present application running on an electronic device (such as Figure 1 the electronic device 10 shown) as an example for illustration. Please refer to Figure 2 As shown, it is a schematic flowchart of the 3D reconstruction method provided by the embodiments of the present application. In one embodiment of the present application, it includes the following steps:

[0042] Step S201, determine the features of the preset object according to the scanning data of the preset object.

[0043] In some embodiments, the preset object may include any object that needs to perform 3D reconstruction, such as a patient's oral cavity, etc. Before determining the features of the preset object according to the scanning data of the preset object, the method further includes: responding to the operation of the user to start scanning, and scanning the preset object in an initial scanning mode to obtain the initial scanning data. The initial scanning mode includes but is not limited to a monocular scanning mode and a binocular scanning mode, and can be set based on system default settings or user preferences.

[0044] In one example, for the convenience of user operation, a button to start scanning can be set in the graphical interaction interface. Responding to the user's selection operation on this button to start scanning, and scanning the preset object in the default initial scanning mode to obtain the initial scanning data. The scanning range corresponding to the initial scanning mode can be a preset range. For example, the preset range can represent the range when scanning at a relatively long distance. In another example, the initial scanning mode can also be determined based on the scanning mode selected by the user. The present application does not make specific limitations on this.

[0045] Among them, if the initial scanning mode is a monocular scanning mode, the electronic device uses a single image sensor to obtain the image data of the preset object. For example, use a camera to take pictures of the preset object from a fixed position of the preset object to obtain the two-dimensional global image data of the preset object. The monocular scanning mode can quickly obtain the overall shape and contour of the preset object, providing a basis for subsequent feature recognition and analysis.

[0046] If the initial scanning mode is a binocular scanning mode, the electronic device uses two image sensors to simulate the stereoscopic vision principle of the human eye to scan a preset object. For example, two cameras simultaneously capture the preset object from different angles, and the depth information of the preset object is obtained by calculating the parallax between the two images. The initial scanning data generated by the binocular scanning mode not only includes the two-dimensional image data of the preset object, but also includes point cloud data or depth maps with three-dimensional spatial information generated by the binocular algorithm, which can provide more detailed and accurate basic data for subsequent three-dimensional reconstruction.

[0047] Based on the above embodiments, the initial scanning mode can be started according to the user operation to quickly scan the preset object, and the initial scanning data of the object can be quickly obtained, laying a foundation for subsequent feature analysis and scanning mode selection.

[0048] In some embodiments, the initial scanning data includes image data. Determining the features of the preset object according to the initial scanning data of the preset object includes: performing semantic segmentation on the preset object according to the image data, and determining multiple different regions of the preset object according to the results of the semantic segmentation; determining the feature point density and depth change rate of each region of the preset object, and determining the features according to the feature point density and the depth change rate.

[0049] In some embodiments, when performing semantic segmentation on the preset object according to the image data, each pixel or region in the image can be classified into a specific semantic category, such as teeth, gums, holes caused by tooth decay, etc., so that the preset object can be divided into multiple different regions according to the semantic categories. In one example, the geometric area of each region can be calculated, for example, the number of pixel points included in each region. In another example, a semantic category label can be set for each region based on the semantic category obtained by semantic segmentation. For example, the semantic category label of a certain region can be a hole.

[0050] In some embodiments, for each region identified by semantic segmentation, its feature point density can be further calculated, such as the number of feature points per unit area. These feature points can be corner points, edge points in the image, or points with significant texture changes. The level of the feature point density can intuitively reflect the richness of the texture features of the region. The depth change rate of each region can also be calculated, for example, by comparing the depth values of adjacent pixels or regions. The depth change rate can reflect the degree of concavity and convexity or structural complexity of the surface of each region. For example, the depth change rate of a hole structure or an uneven surface is larger than that of a plane.

[0051] In some embodiments, the features of each region of a preset object can be determined based on the feature point density and depth change rate of each region of the preset object. For example, if the feature point density of a certain region is high and the depth change rate is also large, then this region may have rich texture features and complex structures; on the contrary, if the feature point density of a certain region is low and the depth change rate is also small, then this region may be relatively smooth and lack significant texture or structural features.

[0052] In some other embodiments, the features of each region can also be determined based on the geometric area and semantic category label of each region of the preset object. Among them, there may be an association relationship between the semantic category label and the depth change rate. For example, the depth change rate of the hole category will be greater than the preset change rate threshold.

[0053] In this way, multiple features of the preset object can be extracted from the initial scan data, and these features have important reference value for subsequent tasks such as scan mode determination or reconstruction.

[0054] Step S202, determine the target scan mode corresponding to different regions of the preset object according to the features.

[0055] In some embodiments, the different regions of the preset object can be divided into multiple types of regions according to the features, and a corresponding target scan mode is assigned to each type of region. In an example, the multiple types of regions include, but are not limited to: the first type of region, the second type of region, the third type of region, and the fourth type of region. The target scan mode includes at least one of the following: a binocular scan mode in a first range, a combined mono-binocular mode in a first range, a binocular scan mode in a second range, a combined mono-binocular mode in a second range. Wherein, the first range is greater than the second range, and the combined mono-binocular mode represents a combined mode of a monocular scan mode and a binocular scan mode.

[0056] In some embodiments, the first type of region represents a region where the geometric area is greater than a preset area threshold, the feature point density is less than or equal to a preset density threshold, and the depth change rate is less than or equal to a preset change rate threshold. Among them, the preset density threshold, the preset density threshold, and the preset change rate threshold can be set according to actual needs, and the present application does not make specific limitations thereon. The first type of region is usually relatively flat and the texture is not complex, and it is suitable for using a large-range binocular scanning mode. Therefore, it can be determined that the target scanning mode corresponding to the first type of region is a binocular scanning mode within a first range. Among them, the first range can represent a relatively large range, and the first range can be smaller than the preset range described in step S201. In this way, the binocular scanning mode within the first range can quickly capture the overall structure of the first type of region, while ensuring the coherence of scanning and the integrity of data. Since the first type of region is usually relatively flat, the binocular scanning mode can accurately calculate the depth information without relying too much on the supplement of monocular scanning.

[0057] In some embodiments, the second type of region represents a region where the geometric area is less than or equal to a preset area threshold, the feature point density is greater than a preset density threshold, and the depth change rate is less than or equal to a preset change rate threshold. Although the area of this type of region may be small, it has rich texture and is suitable for using a small-range binocular scanning mode to capture more details. Therefore, it can be determined that the target scanning mode corresponding to the second type of region is a binocular scanning mode within a second range. Among them, the second range can represent a relatively small range, and the second range can be smaller than the first range. In this way, in the binocular scanning mode within the second range, the scanning range can be accurately controlled, and it can focus on the detailed parts of the second type of region to ensure the accurate capture of texture and microstructures during the scanning process. At the same time, the small-range scanning also helps to reduce unnecessary computational amount and improve the scanning efficiency.

[0058] In some embodiments, the third type of region represents a region where the geometric area is greater than a preset area threshold and the depth change rate is greater than a preset change rate threshold. This type of region has a large area and a complex structure, and a large-range scanning mode combining monocular and binocular can be used to obtain complete scanning data. Therefore, it can be determined that the target scanning mode corresponding to the third type of region is a combined monocular and binocular mode within the first range. In this way, the monocular scanning mode in the combined monocular and binocular mode within the first range can cover a wider third type of region, while the binocular scanning mode can provide accurate depth information in the key areas of the third type of region. This combined mode not only improves the integrity of data but also ensures the accurate reconstruction of complex-structured regions.

[0059] In some embodiments, the fourth type of region represents a region where the geometric area is less than or equal to a preset area threshold and the depth change rate is greater than a preset change rate threshold. Such regions usually have a small area and extremely complex structures (such as deep holes or fine grooves). A small-range combined monocular and binocular scanning mode can be used to improve data integrity and accuracy. Therefore, it can be determined that the target scanning mode corresponding to the fourth type of region is the monocular-binocular combined mode in the second range. In this way, in the monocular-binocular combined mode in the second range, the scanning range is further reduced, enabling in-depth exploration of the fine structures in the fourth type of region, while combining the advantages of monocular and binocular scanning to ensure data accuracy and integrity.

[0060] In some other embodiments, the category of each region can also be determined according to the semantic category of each region, and the corresponding target scanning mode can be assigned to it. For example, it can be determined that a region with a semantic category of hole is the fourth type of region, and the monocular-binocular combined mode in the second range can be assigned to it.

[0061] In some other embodiments, the multi-type regions may further include a fifth type of region. For example, the fifth type of region represents a region where the geometric area is greater than a preset area threshold, the feature point density is greater than a preset density threshold, and the depth change rate is less than or equal to a preset change rate threshold. For the fifth type of region, since it has a large area and complex texture, a scanning mode that can cover a wide area while capturing detailed information can be adopted. For example, it can be determined that the target scanning mode corresponding to the fifth type of region is the binocular scanning mode in the first range to ensure that while capturing the overall structure, the texture details can also be accurately recorded. If the texture is extremely complex or higher precision is required, a local supplementary scan can also be considered by combining the monocular scanning mode. For example, the monocular-binocular combined mode in the first range can be used to further improve data integrity and accuracy.

[0062] In some other embodiments, the multi-type regions may further include a sixth type of region. For example, the sixth type of region represents a region where the geometric area is less than or equal to a preset area threshold, the feature point density is less than or equal to a preset density threshold, and the depth change rate is greater than a preset change rate threshold. For the sixth type of region, since it has a small area but a large depth change, a small-range but high-precision binocular scanning mode can be adopted, and the monocular scanning mode can be combined for local depth measurement. For example, it can be determined that the target scanning mode corresponding to the sixth type of region is the monocular-binocular combined mode in the second range to ensure accurate recording of depth changes within a limited space.

[0063] Based on the above embodiments, different regions of a preset object can be carefully divided based on the characteristics of each region, and corresponding target scanning modes can be intelligently assigned according to characteristics such as the geometric area, feature point density, and depth change rate of each region, which can significantly improve the efficiency and accuracy of 3D reconstruction.

[0064] In some embodiments, different regions can be marked with different identifiers in the interactive interface. For example, the different identifiers can be different numbers, colors, symbols, etc.

[0065] Step S203: In response to the selected target scanning mode of the user, start scanning the corresponding region of the preset object with the selected target scanning mode, and obtain the scanning data of each region of the preset object.

[0066] In some embodiments, all optional target scanning modes can be displayed through a preset interactive interface. A short description or icon can be attached beside each target scanning mode to help the user understand the characteristics and applicable scenarios of each mode. In response to the target scanning mode selected by the user through the interactive interface for scanning a specific region of the preset object, start the selected target scanning mode. For example, in response to the user's selection operation on the icon of the target scanning mode, start the selected target scanning mode.

[0067] In some embodiments, the electronic device starts the target scanning mode selected by the user and configures corresponding scanning parameters, such as scanning speed, resolution, focal length, scanning range, etc. According to the characteristics of the selected target scanning mode, the electronic device automatically adjusts the settings of the scanning device to ensure the best scanning effect. After starting the scanning mode, the electronic device uses the corresponding scanning technology to scan the specified region of the preset object. During the scanning process, the electronic device can display the scanning progress, the visual preview of the scanning region, or the preliminary processing result of the scanning data in real time through the interactive interface. After the scanning is completed, the electronic device collects and stores the scanning data obtained from each region of the preset object. The scanning data can include point cloud data, texture information, depth information, etc., depending on the type of the selected scanning mode and the performance of the scanning device. The obtained scanning data can be optimized, for example, the optimization can include but is not limited to denoising processing, filtering processing, etc., to improve the data quality of the scanning data. In this way, a high-quality scanning data set corresponding to each region can be obtained.

[0068] Based on the above embodiments, it is possible to scan each region of the preset object in the most suitable manner according to the user's selection, solve the contradiction between pursuing scanning details and scanning range in a fixed scanning scenario, avoid the limitation of the scanning range, retain sufficiently good object details while improving the scanning efficiency, so as to obtain high-quality scanning data efficiently and provide a solid foundation for subsequent tasks such as 3D reconstruction.

[0069] Step S204: Fuse the scanning data of all regions of the preset object to obtain a 3D reconstruction model of the preset object.

[0070] In some embodiments, when performing 3D reconstruction on the scan data of all regions of a preset object, if the scan data includes monocular scan data and binocular scan data, the monocular and binocular scan data can be aligned in coordinates, and then the redundant information in the overlapping regions can be identified and removed. Subsequently, global fusion is performed based on the data after removing the overlapping regions. Finally, a global optimization algorithm can be used to deeply process the fused data, thereby accurately constructing a 3D model of the preset object. In one example, the specific implementation steps of step S204 include as follows Figure 3 as shown.

[0071] The 3D reconstruction method provided by the embodiments of the present application accurately identifies the features of the preset object, intelligently selects the most suitable scan mode for each region, and scans each region of the preset object specifically. Finally, the scan data of all regions is seamlessly fused to generate an accurate and complete 3D reconstruction model. It can not only improve the efficiency and accuracy of 3D reconstruction, but also ensure that the model can truly reflect the complex structure and detailed features of the preset object.

[0072] In some embodiments, referring to Figure 3 as shown, the specific implementation steps of step S204 include the following process:

[0073] Step S301, if the scan data includes monocular scan data and binocular scan data, align the coordinates of the monocular scan data and the binocular scan data.

[0074] In some embodiments, if the scan data contains monocular scan data and binocular scan data, the coordinates of these two types of data can be aligned, so as to convert the data obtained under different scan modes into a unified coordinate system, effectively ensuring the spatial consistency of the data. In one example, the algorithms used for coordinate alignment may include but are not limited to feature point matching, Iterative Closest Point (ICP) algorithm or other related algorithms.

[0075] In some embodiments, through coordinate alignment, not only can the monocular scan data and the binocular scan data be converted into a unified coordinate system, but also the scan data obtained due to differences in scan ranges in different regions can be integrated into this unified coordinate system. In this way, it can be ensured that the data from different scan sources, different scan positions, and even different scan time points can be compared and analyzed in the same spatial framework, enabling subsequent 3D reconstruction to be based on a coherent and unambiguous data set and constructing a more accurate 3D model.

[0076] Step S302, determine the overlapping regions of the monocular scan data and the binocular scan data after coordinate alignment, and remove the redundant data in the overlapping regions.

[0077] In some embodiments, after coordinate alignment is completed, the overlapping regions of the monocular scan data and the binocular scan data can be identified in a unified coordinate system, and the redundant data in these overlapping regions can be removed. In one example, based on the spatial geometric relationship or image matching algorithm between the monocular scan data and the binocular scan data, the data points in the monocular scan data and the binocular scan data that are less than a preset distance threshold in spatial distance can be determined, and such data points are used as the redundant data in the overlapping region. When removing the redundant data, data points with higher precision and richer information, such as the points in the binocular scan data, can be preferentially selected. In this way, it can help reduce data redundancy and improve the accuracy and efficiency of 3D reconstruction.

[0078] Step S303: Based on the scan data obtained after removing redundant data from all regions, perform global data fusion to obtain global fusion data.

[0079] In some embodiments, global data fusion refers to effectively integrating the information in multiple data sources to form a more comprehensive and accurate data set. In the field of 3D reconstruction, global data fusion generally includes stitching and fusing data such as point clouds, meshes, or voxels in multiple scan data sets to form a complete 3D model.

[0080] In some embodiments, during global data fusion, the scan data after removing redundant data from all regions can be globally stitched according to the positions of the preset landmark points of the preset object to obtain the global fusion data. In one example, the landmark points can be points with clear geometric features, such as corner points, edge points, etc., or specific objects or structures determined through semantic recognition, such as the corner points of teeth and the edge points of gums in a patient's oral cavity. By determining the positions of these landmark points in each scan data set, precise alignment and stitching between the scan data can be achieved.

[0081] In one example, the landmark points can be extracted from each scan data set through an image processing algorithm or a machine learning algorithm. For example, the image processing algorithm can include, but is not limited to, algorithms such as corner detection and edge detection, and the machine learning algorithm can include, but is not limited to, algorithms such as feature matching and deep learning. Then, by calculating the spatial distance, angular relationship, etc. between the landmark points, these landmark points are used to match between the scan data sets to ensure precise alignment between the scan data sets. After completing the landmark point matching, the transformation matrix of the same landmark point in different scan data sets, such as the rotation matrix, translation matrix, etc., can be calculated, and the scan data sets are stitched using the transformation matrix to ensure seamless connection between the scan data in different regions and obtain the global fusion data.

[0082] Step S304: Perform global optimization on the global fusion data to obtain the 3D reconstruction model.

[0083] In some embodiments, the globally fused data may still contain some errors, inconsistencies, or overlapping regions. The globally fused data can be further globally optimized and processed to eliminate these potential problems and finally generate a high-quality 3D reconstruction model.

[0084] In some embodiments, there may still be overlapping regions between the scanned data of the globally fused data. For example, the overlapping regions caused by the edges of the scanned data sets in different regions after splicing. These overlapping regions may cause redundancy or errors in the 3D model. Therefore, an overlapping removal operation can be performed on the globally fused data, and the binocular scanned data can be retained during the overlapping removal operation. In this way, there are no overlapping regions between the edges of different regions in the globally fused data after the overlapping removal operation, which can improve the accuracy of the 3D model.

[0085] In some embodiments, after the overlapping removal operation, there may be problems such as discontinuous edges or unnatural transitions between the scanned data of different regions. A uniform point distance transition algorithm can be used to perform a smoothing transition process on the edges of the scanned data of different regions. Among them, the uniform point distance transition algorithm can determine the parameters and algorithms required for the smoothing transition by calculating the point cloud density and geometric features between the scanned data of different regions. In this way, the edge continuity and transition naturalness between the scanned data of different regions can be improved, and the accuracy of the 3D model can be improved.

[0086] In some embodiments, after performing the above optimizations, error correction and overall optimization can also be performed on the globally fused data. For example, the globally fused data can be error-corrected and overall optimized by comparing the spatial position relationships, geometric features, etc. between different scanned data. Among them, error correction methods such as the least squares method and the iterative closest point algorithm can be used to eliminate the errors and inconsistencies between the scanned data. In addition, overall optimization methods such as mesh optimization and surface reconstruction can also be used to improve the accuracy and integrity of the 3D reconstruction model.

[0087] Based on the above embodiments, through the coordinate alignment of monocular and binocular scanned data, the removal of redundant data in the overlapping regions, global data fusion, and subsequent global optimization processing, the data of different scanning modes and regions can be integrated into a unified coordinate system, and data redundancy, errors, and inconsistencies can be effectively eliminated. In this way, it can be ensured that the 3D reconstruction model can be constructed based on a coherent and unambiguous data set, thereby significantly improving the accuracy, integrity, and natural transition of the 3D model, and providing a high-quality 3D visualization solution for various application scenarios.

[0088] Figure 4 It is a structural diagram of a 3D reconstruction device provided by an embodiment of the present application.

[0089] In some embodiments, the three-dimensional reconstruction device 40 may include a plurality of functional modules composed of computer program segments. The computer programs of each program segment in the three-dimensional reconstruction device 40 may be stored in the memory of the electronic device and executed by at least one processor to perform the functions of three-dimensional reconstruction (see details in Figure 2 description).

[0090] In this embodiment, the three-dimensional reconstruction device 40 can be divided into a plurality of functional modules according to the functions it performs. The functional modules may include: a determination module 401, an acquisition module 402, and a reconstruction module 403. The module referred to in this application means a series of computer program segments that can be executed by at least one processor and can complete fixed functions, and are stored in the memory. In this embodiment, for the implementation manners of the functions of each module in the three-dimensional reconstruction device 40, reference may be made to the limitations on the three-dimensional reconstruction method above, and details will not be repeated here.

[0091] The determination module 401 is configured to determine the features of the preset object according to the initial scan data of the preset object.

[0092] The determination module 401 is further configured to determine the target scan mode corresponding to different regions of the preset object according to the features.

[0093] The acquisition module 402 is configured to, in response to the target scan mode selected by the user, start scanning the corresponding region of the preset object in the selected target scan mode, and acquire the scan data of each region of the preset object.

[0094] The reconstruction module 403 is configured to fuse the scan data of all regions of the preset object to obtain a three-dimensional reconstruction model of the preset object.

[0095] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and the method implemented when the program instructions are executed may refer to the methods in the above embodiments of the present application.

[0096] Among them, the computer-readable storage medium may be the internal memory of the electronic device in the above embodiment, such as the hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc.

[0097] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area. The program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0098] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0099] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0100] In the embodiments provided in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0101] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Determining the features of the preset object according to the initial scan data of the preset object; Determining the target scan modes corresponding to different regions of the preset object according to the features, including: Dividing different regions of the preset object into multiple types of regions according to the features, where the multiple types of regions include a first type of region and a fourth type of region. Among them, the first type of region represents a region with a geometric area greater than a preset area threshold and a depth change rate less than or equal to a preset change rate threshold, and the fourth type of region represents a region with a geometric area less than or equal to the preset area threshold and a depth change rate greater than the preset change rate threshold; Determining that the target scan mode corresponding to the first type of region is a binocular scan mode within a first range, and determining that the target scan mode corresponding to the fourth type of region is a combined single-binocular mode within a second range, where the first range is greater than the second range, and the combined single-binocular mode represents a combined mode of a monocular scan mode and a binocular scan mode; Responding to the target scan mode selected by the user, starting the selected target scan mode to scan the corresponding region of the preset object, and obtaining the scan data of each region of the preset object; Fusing the scan data of all regions of the preset object to obtain a three-dimensional reconstruction model of the preset object.

2. The three-dimensional reconstruction method according to claim 1, wherein Before determining the features of the preset object according to the scan data of the preset object, the method further includes: Responding to the operation of starting the scan performed by the user, scanning the preset object using an initial scan mode to obtain the initial scan data, where the initial scan mode includes a monocular scan mode and a binocular scan mode.

3. The three-dimensional reconstruction method according to claim 1, wherein The initial scan data includes image data, and determining the features of the preset object according to the initial scan data of the preset object includes: Performing semantic segmentation on the preset object according to the image data, and determining multiple different regions of the preset object according to the results of the semantic segmentation; Determining the feature point density and depth change rate of each region of the preset object, and determining the features according to the feature point density and the depth change rate.

4. The three-dimensional reconstruction method according to claim 1, characterized in that The determining the target scan modes corresponding to different regions of the preset object according to the features further includes: Dividing different regions of the preset object into multiple types of regions according to the features, where the multiple types of regions include a second type of region and a third type of region. Among them, the second type of region represents a region with a geometric area less than or equal to the preset area threshold and a depth change rate less than or equal to the preset change rate threshold, and the third type of region represents a region with a geometric area greater than the preset area threshold and a depth change rate greater than the preset change rate threshold; Determining that the target scan mode corresponding to the second type of region is a binocular scan mode within the second range; Determining that the target scan mode corresponding to the third type of region is a combined single-binocular mode within the first range.

5. The three-dimensional reconstruction method according to claim 4, wherein The first type of region also represents a region with a geometric area greater than a preset area threshold, a feature point density less than or equal to a preset density threshold, and a depth change rate less than or equal to a preset change rate threshold; The second type of region also represents a region where the geometric area is less than or equal to a preset area threshold, the feature point density is greater than a preset density threshold, and the depth change rate is less than or equal to a preset change rate threshold.

6. The three-dimensional reconstruction method according to claim 1, wherein The fusion of the scan data of all regions of the preset object to obtain the three-dimensional reconstruction model of the preset object includes: If the scan data includes monocular scan data and binocular scan data, align the coordinates of the monocular scan data and the binocular scan data. Determine the overlapping region of the monocular scan data and the binocular scan data after coordinate alignment, and remove redundant data in the overlapping region. Based on the scan data obtained after removing redundant data from all regions, perform global data fusion to obtain global fusion data. Perform global optimization on the global fusion data to obtain the three-dimensional reconstruction model.

7. The three-dimensional reconstruction method according to claim 6, characterized in that The performing global data fusion based on the scan data obtained after removing redundant data from all regions to obtain global fusion data includes: According to the positions of the preset landmark points of the preset object, perform global stitching on the scan data after removing redundant data from all regions to obtain the global fusion data.

8. The three-dimensional reconstruction method according to claim 6, wherein The optimization of the global fusion data includes: Perform a de-overlapping operation on the global fusion data and retain the binocular scan data in the de-overlapping operation. Adopt a uniform point distance transition algorithm to perform smooth transition processing on the edges of the scan data in different regions.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory storing one or more application programs, the one or more application programs being configured to implement the three-dimensional reconstruction method according to any one of claims 1 to 8 when executed by the processor.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by the processor of the electronic device, implements the three-dimensional reconstruction method according to any one of claims 1 to 8.

Images