Control method and apparatus for three-dimensional scanning device, terminal device, and system

Abstract

The present disclosure provides a control method and apparatus for a three-dimensional scanning device, a terminal device, and a system. The method comprises: acquiring a target image collected by the three-dimensional scanning device; determining, on the basis of the target image, the current scanning environment type of the three-dimensional scanning device; switching the current operating mode of the three-dimensional scanning device to a target operating mode that matches the scanning environment type, or prompting a user to switch the current operating mode of the three-dimensional scanning device to the target operating mode that matches the scanning environment type, wherein the three-dimensional scanning device is configured with at least two operating modes, and each of operating modes corresponds to one scanning environment type; when the three-dimensional scanning device is in different operating modes, in the process of using the image collected by the three-dimensional scanning device to perform three-dimensional reconstruction on a target object, the image is processed in different manners, and / or in the process of using the three-dimensional scanning device to collect the image, components in the three-dimensional scanning device operate in different manners.

Description

Control method, device, terminal device and system for three-dimensional scanning equipment

[0001] Cross-reference

[0002] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on December 14, 2023, with application number 202311729669.X and entitled “Control method, device, terminal device and system for three-dimensional scanning equipment,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of three-dimensional scanning technology, and in particular to a control method, device, terminal device, and system for three-dimensional scanning equipment. Background Art

[0004] With the development of 3D scanning technology, it has been widely used in various fields such as medicine, industry, and archaeology to achieve three-dimensional reconstruction of target objects. When users use 3D scanning equipment to scan target objects, in order to obtain ideal reconstruction effects, usually, for different types of scanning environments, there are differences in the way images are processed when the images collected by the 3D equipment are subsequently used for three-dimensional reconstruction. At present, when switching the working mode of the 3D scanning device, it is mainly controlled manually by the user. It is easy for the user to forget to switch the working mode, resulting in inappropriate image processing in the current scanning environment, resulting in poor 3D reconstruction effects and the need to re-enter or re-scan. Summary of the Invention

[0005] The present disclosure provides a control method, device, terminal device and system for a three-dimensional scanning device.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for controlling a three-dimensional scanning device is provided, the method comprising:

[0007] Acquire a target image captured by a three-dimensional scanning device;

[0008] determining a current scanning environment type of the three-dimensional scanning device based on the target image;

[0009] Switching the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type, or prompting the user to switch the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type;

[0010] Among them, the three-dimensional scanning device is configured with at least two working modes, each working mode corresponds to a scanning environment type. When the three-dimensional scanning device is in different working modes, the image is processed differently in the process of three-dimensionally reconstructing the target object using the image captured by the three-dimensional scanning device, and / or, in the process of capturing the image using the three-dimensional scanning device, the operation mode of each component in the three-dimensional scanning device is different.

[0011] According to a second aspect of an embodiment of the present disclosure, a control device for a three-dimensional scanning device is provided, the control device comprising:

[0012] An acquisition module, configured to acquire a target image captured by a three-dimensional scanning device;

[0013] a scanning environment type determination module configured to determine a current scanning environment type of the three-dimensional scanning device based on the target image;

[0014] a processing module configured to switch a current operating mode of the three-dimensional scanning device to a target operating mode that matches the type of scanning environment, or to prompt a user to switch the current operating mode of the three-dimensional scanning device to the target operating mode that matches the type of scanning environment;

[0015] Among them, the three-dimensional scanning device is configured with at least two working modes, each working mode corresponds to a scanning environment type. When the three-dimensional scanning device is in different working modes, the image is processed differently in the process of three-dimensionally reconstructing the target object using the image captured by the three-dimensional scanning device, and / or, in the process of capturing the image using the three-dimensional scanning device, the operation mode of each component in the three-dimensional scanning device is different.

[0016] According to a third aspect of an embodiment of the present disclosure, a terminal device is provided, which includes a processor, a memory, and computer instructions stored in the memory for execution by the processor. When the processor executes the computer instructions, the control method of the three-dimensional scanning device mentioned in the first aspect above can be implemented.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a control system for a three-dimensional scanning device is provided, the control system including the terminal device mentioned in the third aspect above, and a three-dimensional scanning device communicatively connected to the terminal device, the three-dimensional scanning device being configured to capture images of the area to be scanned and send the images to the terminal device.

[0018] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the control method of the three-dimensional scanning device mentioned in the first aspect is implemented.

[0019] In the embodiment of the present disclosure, the current scanning environment type of the three-dimensional scanning device can be automatically determined based on the image collected by the three-dimensional scanning device, and then the working mode of the three-dimensional scanning device can be automatically switched to a target working mode that is adapted to the scanning environment type, or the user can be prompted to switch the working mode of the three-dimensional scanning device to a target working mode that is adapted to the scanning environment type, thereby avoiding the problem that the user forgets to set the working mode, resulting in poor effect of the final reconstructed three-dimensional model, and the need for re-three-dimensional reconstruction or re-scanning.

[0020] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0022] FIG1 is a schematic diagram of an application scenario of an embodiment of the present disclosure.

[0023] FIG2 is a schematic diagram of configuring multiple working modes for a three-dimensional scanning device according to an embodiment of the present disclosure.

[0024] FIG3 is a flow chart of a method for controlling a three-dimensional scanning device according to an embodiment of the present disclosure.

[0025] FIG4 is a schematic diagram illustrating the difference between the processing methods of an intraoral working mode and an extraoral working mode according to an embodiment of the present disclosure.

[0026] FIG5 is a schematic diagram of a detection result of a scanning environment type obtained based on a target image according to an embodiment of the present disclosure.

[0027] FIG6 is a schematic diagram of the logical structure of a control device of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0028] FIG7 is a schematic diagram of the logical structure of a terminal device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0030] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0031] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0032] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure are further described in detail below with reference to the accompanying drawings.

[0033] With the development of 3D scanning technology, it has been widely used in various fields, such as medicine, industry, and archaeology, to achieve three-dimensional reconstruction of target objects. When users use 3D scanning equipment to scan target objects, in order to achieve ideal reconstruction results, the components on the 3D scanning equipment usually operate differently when capturing images for different scanning environments. This can also lead to differences in the way the images are processed when the 3D equipment is used for 3D reconstruction.

[0034] For example, to adapt to different scanning environments, 3D scanning equipment can be configured with multiple operating modes. When acquiring images in these modes, the various components within the 3D scanning equipment can be configured to operate in appropriate ways. Alternatively, when performing 3D reconstruction using images captured by the 3D device, appropriate processing methods can be selected to achieve the best possible 3D reconstruction results.

[0035] Currently, switching the working mode of a 3D scanning device is primarily done manually by the user. For example, a mode switching button can be set on the 3D scanning device. Before scanning, the user can use this mode switching button to switch the working mode of the 3D scanning device to the appropriate working mode based on the type of scanning environment. However, in many scenarios, the user may forget to perform the working mode setting operation, or set the working mode incorrectly, resulting in poor results in the final reconstruction of the target object, requiring re-3D reconstruction or even re-scanning, which seriously affects scanning efficiency and user experience.

[0036] For example, taking the application of 3D scanning equipment in oral scanning scenarios as an example, when reconstructing a tooth model, the user may need to scan the real teeth inside the mouth, or may need to scan the tooth model outside the mouth. This is because there are large differences between the intraoral environment and the extraoral environment. For example, there are interferences such as gums, tongue, and buccal soft tissue in the intraoral environment, while the extraoral environment does not have the above problems, or the brightness in the intraoral environment is lower than that in the extraoral environment. Therefore, for the two scanning environments, there are differences in the way the images are processed when the images are subsequently used to reconstruct teeth. Therefore, it is necessary to set different working modes for the above two usage scenarios and configure appropriate processing methods for each working mode. However, when using it, users usually forget to switch the working modes, resulting in poor results of the final reconstructed three-dimensional tooth model, which requires re-three-dimensional reconstruction or re-scanning.

[0037] Based on this, an embodiment of the present disclosure provides a control method for a three-dimensional scanning device, which can automatically determine the current scanning environment type of the three-dimensional scanning device based on the image collected by the three-dimensional scanning device, and then automatically switch the working mode of the three-dimensional scanning device to a target working mode that is adapted to the scanning environment type, or prompt the user to switch the working mode of the three-dimensional scanning device to a target working mode that is adapted to the scanning environment type, thereby avoiding the problem that the user forgets to set the working mode, resulting in poor effect of the final reconstructed three-dimensional model, and the need for re-three-dimensional reconstruction or re-scanning.

[0038] The control method of the three-dimensional scanning device provided in the embodiment of the present disclosure can be executed by the three-dimensional scanning device, or by a terminal device connected to the three-dimensional scanning device wirelessly or wiredly. The terminal device can be a tablet, a laptop, etc., or some steps can be executed by the three-dimensional scanning device, and some steps can be executed by the terminal device.

[0039] For example, in some scenarios, if the 3D scanning device has sufficient computing power, it can capture images and use them to perform 3D reconstruction, resulting in a 3D reconstructed model. Therefore, the 3D scanning device can switch the operating mode of its components and the processing method for 3D reconstruction based on the scanning environment type.

[0040] In some scenarios, if the 3D scanning device's own computing power is limited, 3D reconstruction may require the assistance of a terminal device. For example, the terminal device could be installed with scanning software that performs real-time 3D reconstruction based on images captured by the 3D scanning device and displays the reconstruction results to the user. The 3D scanning device could then switch the operating mode of its components based on the scanning environment, and the scanning software could switch the processing method for 3D reconstruction of the image.

[0041] For example, as shown in FIG1 , which is a schematic diagram of an application scenario of an embodiment of the present disclosure, considering that the computing power of a three-dimensional scanning device is relatively weak, the images collected by the three-dimensional scanning device are usually sent to a terminal device with better processing performance that is connected to the terminal device. The terminal device can be installed with scanning software, which performs real-time three-dimensional reconstruction based on the collected images and displays the reconstruction results to the user. At the same time, the scanning software can detect the type of scanning environment based on the collected images and send a mode switching instruction to the three-dimensional scanning device based on the detection results to control the three-dimensional scanning device to switch modes. For example, the three-dimensional scanning device can be controlled to switch the operating mode of each component, and the three-dimensional scanning software can also switch its own processing method for three-dimensional reconstruction of the image.

[0042] To adapt to different types of scanning environments, as shown in Figure 2, a 3D scanning device can be configured with multiple operating modes based on its actual usage scenario, where each operating mode corresponds to a specific type of scanning environment. For each operating mode, the operating mode of each component in the 3D scanning device when capturing images using the 3D scanning device, and / or the processing mode for images captured by the 3D scanning device when performing 3D reconstruction, can be pre-set. In some embodiments of the present disclosure, the operating mode of the 3D scanning device can include the operating mode of each component in the 3D scanning device and the processing mode of the 3D scanning software compatible with the 3D scanning device.

[0043] For example, in order to achieve the best reconstruction results for different scanning environments, the image processing method used during the 3D reconstruction process is determined. The processing method includes the processing steps involved in image processing, the processing algorithm used, the processing parameters used, and so on. The processing methods for different working modes can be pre-configured. When switching to a certain working mode, the corresponding processing method can be automatically called to process the image to complete the 3D reconstruction.

[0044] For example, in order to obtain higher-quality images in different scanning environments, the operating modes of the components on the 3D scanning device may vary. These operating modes can refer to the operating parameters or states of various components within the 3D scanning device, such as the status of the fill light, the brightness of the fill light, the status of the anti-fog module, or the intensity of the projected structured light.

[0045] A mode switching button may be provided on the 3D scanning device, as shown in FIG2 , to switch between different working modes, or the user may switch the working mode of the 3D scanning device through scanning software.

[0046] In addition, devices such as the three-dimensional scanning device or the terminal device may also be provided with an operating mode indicator light to indicate the current operating mode of the three-dimensional scanning device.

[0047] FIG3 is a flow chart of a control method for the three-dimensional scanning device, which may include the following steps:

[0048] S302, obtaining a target image captured by a three-dimensional scanning device;

[0049] In step S302, a target image captured by the 3D scanning device is obtained. The target image is used to detect the type of scanning environment currently being scanned by the 3D scanning device. To improve the accuracy of the detection results, the target image may be one or more frames selected from images captured by the 3D scanning device that are suitable for scanning environment detection.

[0050] Among them, the three-dimensional scanning equipment can be an oral scanner, a facial scanner, an industrial scanner, or a professional scanner, which can be used for three-dimensional reconstruction of teeth, faces, bodies, industrial products, industrial equipment, cultural relics, artworks, prostheses, medical devices, buildings and other objects.

[0051] S304, determining the current scanning environment type of the three-dimensional scanning device based on the target image;

[0052] In step S304, the current scanning environment type of the 3D scanning device can be determined based on the target image. For example, a neural network model or a deep neural network model can be used to detect the target image. For example, a model for detecting the scanning environment type can be pre-trained and then used to detect the scanning environment type. Alternatively, other methods can be used to detect the target image to determine the current scanning environment type of the 3D scanning device.

[0053] S306: Switch the current working mode of the 3D scanning device to a target working mode that matches the scanning environment type, or prompt the user to switch the current working mode of the 3D scanning device to a target working mode that matches the scanning environment type.

[0054] In step S306, after determining the current scanning environment type of the three-dimensional scanning device, the current working mode of the three-dimensional scanning device can be automatically switched to a target working mode that matches the scanning environment type. For example, the correspondence between the scanning environment type and the working mode has been pre-configured, so the target working mode can be determined directly based on the correspondence, and the three-dimensional scanning device can be controlled to switch to the target working mode. For example, the scanning software can send a mode switching instruction to the three-dimensional scanning device to control the three-dimensional scanning device to automatically switch to the target mode. In some scenarios, after determining the target working mode that matches the current scanning environment type, a prompt message can also be sent to the user, prompting the user to switch the current working mode of the three-dimensional scanning device to the target working mode. For example, the user can be prompted to switch the current working mode to the target working mode through voice prompts, text prompts, pop-up prompts, ringtone prompts, vibration prompts, etc.

[0055] After the three-dimensional scanning device is switched to the target working mode, each component in the three-dimensional scanning device can operate in accordance with the operating mode corresponding to the target working mode, and / or when three-dimensional reconstruction is performed using images captured by the three-dimensional scanning device, the image can be processed using the processing method corresponding to the target working mode to reconstruct a three-dimensional model of the target object.

[0056] Through the solution provided by the embodiment of the present disclosure, the scanning environment type of the three-dimensional scanning device can be automatically detected, and the working mode that matches the current scanning environment type can be automatically determined, and then the user can be prompted to switch to the working mode, or the working mode can be automatically switched to the working mode, thereby avoiding the problem that the working mode and the scanning environment type do not match due to the user forgetting to set the working mode, resulting in poor reconstructed three-dimensional model effect.

[0057] When scanning a target object for 3D reconstruction, two scenarios exist: one is when there are relatively few interfering objects around the target object, and the image captured during the scanning process contains relatively few impurities, primarily the target object. In this scenario, the captured image can be used directly for 3D reconstruction. The other is when there are relatively many interfering objects around the target object, and the image captured during the scanning process contains relatively many impurities. If the captured image is used directly for subsequent 3D reconstruction, the interference from the impurities will result in low accuracy and poor results in the reconstructed 3D model. Therefore, in this scenario, during the 3D reconstruction process, data in the image that is not relevant to the 3D reconstruction of the target object can be first identified and then marked. This data is then not required to participate in the subsequent 3D reconstruction, thereby improving the accuracy of the reconstructed 3D model. Therefore, in some embodiments of the present disclosure, different processing methods can refer to whether the processing steps for processing the image include a target step, wherein the target step is configured to identify data in the image that is not relevant to the target object to be reconstructed, so that this irrelevant data is not used in the subsequent 3D reconstruction of the target object.

[0058] For example, in some embodiments of the present disclosure, the 3D scanning device is an oral 3D scanning device. When scanning teeth to reconstruct a 3D model of the teeth, the oral 3D scanning device operates in two scenarios: one is scanning a user's actual teeth within the oral cavity. In this case, the captured image contains a significant amount of impurities due to the numerous distracting objects surrounding the teeth, such as the tongue, buccal soft tissue, and gums. The other is scanning a tooth model outside the oral cavity, such as a tooth model made of paraffin, metal, or resin. In this case, the captured image contains fewer impurities due to the absence of distracting objects surrounding the teeth. Therefore, as shown in FIG4 , the scanning environment type can be divided into an intraoral environment and an extraoral environment. The operating mode can include an intraoral operating mode corresponding to the intraoral environment and an extraoral operating mode corresponding to the extraoral environment. When the operating mode is the intraoral operating mode, the processing steps of the image captured by the 3D scanning device during the 3D reconstruction process include the target step described above. When the operating mode is the extraoral operating mode, the processing steps do not include the target step described above. Among them, data that is not related to the three-dimensional reconstruction of the target object may refer to areas such as the tongue, buccal soft tissue, and part of the gums in the image. The image data corresponding to these areas in the image can be deleted first, and then the subsequent three-dimensional reconstruction processing flow can be carried out to avoid interference of these data with the three-dimensional reconstruction.

[0059] In some embodiments of the present disclosure, different processing methods for processing images captured by a three-dimensional scanning device may refer to different processing parameters when processing these images, wherein the processing parameters may include one or more of the following: the brightness enhancement amplitude when the brightness of the image is enhanced, and the maximum stitching error allowed when the image is stitched. For example, in the process of three-dimensional reconstruction using images captured by a three-dimensional scanning device, considering that the ambient brightness of some scenes is darker, the brightness of the image can be enhanced to improve the display effect of the reconstructed three-dimensional model. The brightness of the scanning environment is different for different scenes. Therefore, when enhancing the brightness of the image, different brightness enhancement amplitudes can be set for different scanning environment types to adjust the image brightness to an appropriate brightness.

[0060] In addition, in the process of three-dimensional reconstruction using images collected by a three-dimensional scanning device, it is necessary to stitch together multiple frames of images continuously collected by the three-dimensional scanning device to obtain a complete target object. For different scanning environments, in order to achieve successful image stitching, there are certain differences in the maximum stitching error allowed when stitching images. For example, in some scenarios, the target object is a rigid object, and its shape and position will not change during the scanning process. In this case, in order to ensure accurate stitching, the stitching error can be set to a smaller value. In some scenarios, the target object is not a rigid object, and its shape and position may change slightly during the scanning process. In this case, in order to ensure that the same area can be successfully stitched together, the stitching error can be set to a larger value. Therefore, different maximum stitching errors can be set according to the characteristics of the target object being scanned in different types of scanning environments.

[0061] In some embodiments of the present disclosure, as shown in FIG4 , the three-dimensional scanning device is an oral three-dimensional scanning device, the scanning environment types include an intraoral environment and an extraoral environment, and the working modes include an intraoral working mode corresponding to the intraoral environment and an extraoral working mode corresponding to the extraoral environment. Considering the scene of intraoral scanning, since the ambient brightness is often low, when enhancing the brightness of the image, the enhancement amplitude should be set to a larger value, while in the scene of extraoral scanning, since the ambient brightness is high, when enhancing the brightness of the image, the enhancement amplitude should be set to a smaller value. Therefore, when the working mode is the intraoral working mode, the brightness enhancement amplitude can be set to a first amplitude, and when the working mode is the extraoral working mode, the brightness enhancement amplitude can be set to a second amplitude, wherein the first amplitude is greater than the second amplitude. By setting different brightness enhancement amplitudes for different scanning environment types, it is possible to ensure that the brightness of the processed image is adjusted to an appropriate value, thereby ensuring the display effect of the processed image.

[0062] In addition, considering the scene of intraoral scanning, due to the presence of non-rigid objects such as gums and cheeks, the shape and position of the teeth may change slightly, which will affect the smoothness and accuracy of the image during the stitching process. The scene of extraoral scanning does not have the above problem. Therefore, in the scene of intraoral scanning, the maximum stitching error allowed when stitching images can be set larger to ensure that the images can be successfully stitched. In the scene of extraoral scanning, the maximum stitching error allowed when stitching images can be set smaller to ensure stitching accuracy. Therefore, as shown in Figure 4, in some embodiments of the present disclosure, when the working mode is the intraoral working mode, the maximum stitching error can be set to the first stitching error, and when the working mode is the extraoral working mode, the maximum stitching error can be set to the second stitching error, wherein the first stitching error is greater than the second stitching error. By setting different stitching errors for different scanning environment types, it can be ensured that images can be successfully and accurately stitched in different scanning scenarios.

[0063] Furthermore, different operating modes of the three-dimensional scanning device may mean that the operating modes of various components in the three-dimensional scanning device are different. For example, when the three-dimensional scanning device of the three-dimensional scanning device is an oral scanner, the operating modes of various components may include whether the fill light module is running and whether the anti-fog module is turned on. For example, when the operating mode is the intraoral working mode, the anti-fog module and the fill light module are turned on, and when the operating mode is the extraoral working mode, the anti-fog module and the fill light module are turned off. The anti-fog module may include a heating element, which is configured to prevent the lenses of the three-dimensional scanning device from fogging. The fill light module may include a fill light, which is configured to fill light the acquisition range of the three-dimensional scanning device.

[0064] When the 3D scanning device is a handheld scanner, the operating modes of various components vary, including whether the laser speckle module, line laser module, stripe light module, phase-shifted stripe module, visible light module, and invisible light module are operating. For example, when operating in human mode, the visible light module is turned off and the invisible light module is turned on. When operating in industrial mode, the visible light module is turned on and the invisible light module is turned off.

[0065] For example: Consider that even in the intraoral scanning scenario, there are differences. For example, some patients are older and most of their teeth have fallen out, while some patients still have intact teeth. For scenarios with different tooth loss conditions, when processing images to achieve three-dimensional reconstruction, the image processing method can also be adaptively adjusted to obtain more accurate three-dimensional reconstruction results. Therefore, in some embodiments of the present disclosure, the intraoral working mode can be further subdivided into at least two sub-modes, wherein the tooth loss conditions in the scanned oral cavity are different in different sub-modes, and the image processing methods in different sub-modes are different. For example, the intraoral working mode can be subdivided into an edentulous jaw mode and a dentate jaw mode, and the processing methods can be adaptively adjusted for these two modes respectively.

[0066] Specifically, in both the edentulous and dentate models, when determining data that is irrelevant to the 3D reconstruction of the target object (i.e., teeth) to be reconstructed, the criteria for determining this data will be adjusted. For example, in the edentulous model, all buccal and lingual data will be treated as irrelevant data, while in the edentulous model, the lingual data and the buccal data located further outward will be treated as irrelevant data. However, a portion of the buccal data located further inward, as well as more dental arch data and gum data, will be used as 3D reconstruction-related data for subsequent 3D reconstruction. Furthermore, if the edentulous model is used for a full denture medical scenario, more buccal or gum data can be retained for subsequent 3D reconstruction.

[0067] Similarly, the extraoral scanning scene is similar, because the extraoral scanning scene is mainly to scan tooth models, and tooth models often include a variety of materials, such as plaster, wax embankment, metal, resin, etc. When performing three-dimensional reconstruction on tooth models of different materials, the image processing method can also be adaptively adjusted to obtain more accurate three-dimensional reconstruction results. Therefore, in some embodiments of the present disclosure, the extraoral working mode can be further subdivided into multiple sub-modes, and the materials of the scanned tooth models in different sub-modes are different, and the image processing methods in different sub-modes are different. For example, taking a tooth model made of metal as an example, the brightness of the structured light projected by the three-dimensional scanning equipment can be enhanced, and more small block data can be retained during subsequent three-dimensional reconstruction, making the reconstructed three-dimensional model more complete.

[0068] In some embodiments of the present disclosure, in order to ensure the accuracy of the detection results of the current scanning environment type of the three-dimensional scanning device, the target image can be a multi-frame image, and each frame of the target image in the multi-frame target image can be detected separately to determine the current scanning environment type of the three-dimensional scanning device and obtain the detection result corresponding to the frame target image. If the detection results of these multiple frames of target images are consistent, the detection results of these multiple frames of target images are used as the current scanning environment type of the three-dimensional scanning device. For example, as shown in Figure 5, a detection number threshold can be set, assuming it is 4 times, and then 4 frames of target images can be obtained from the images collected by the three-dimensional scanning device respectively, and the current scanning environment type is determined based on each frame of the target image. If the detection results of 4 consecutive times are consistent, for example, they are all intraoral environments, then the current detection result (intraoral environment) can be used as the current scanning environment type of the three-dimensional scanning device. By determining the final detection result based on the detection results of multiple frames of images, the accuracy of the detection result can be improved.

[0069] In some embodiments of the present disclosure, considering that the images continuously captured by a 3D scanning device may all be images of the same area within the scanning environment, i.e., the image content is substantially the same, if multiple target image frames correspond to the same area, then the detection results used to determine the current scanning environment type of the 3D scanning device based on these multiple target image frames will inevitably be consistent, which does not help improve the accuracy of the detection results. Therefore, when acquiring target images, one frame can be captured at intervals of a preset duration, thereby ensuring a longer acquisition time interval between adjacent target image frames, i.e., the adjacent target image frames correspond to different areas within the scanning environment. In this way, the detection results determined based on these multiple target image frames are more accurate.

[0070] In some embodiments of the present disclosure, when detecting the current scanning environment type of a 3D scanning device, it is important to consider that not all images captured by the 3D scanning device are suitable for detecting the scanning environment type. For example, in the case of scanning a patient's oral cavity, after the user clicks the "Start Scan" button, they may not immediately place the oral 3D scanning device inside the patient's oral cavity for scanning. This means that the image captured by the 3D scanning device at this time is not an image of the actual scanning scene, i.e., an invalid image. Consequently, the results of detecting the scanning environment type using such an image are unreliable. Therefore, to improve detection efficiency, the target image used for detecting the scanning environment type can be a valid image selected from the images captured by the 3D scanning device. A valid image is an image that can be successfully spliced ​​with other images during the 3D reconstruction process to reconstruct a 3D model of the target object. In some embodiments of the present disclosure, to improve the efficiency of selecting valid images, during the 3D reconstruction process using images captured by the 3D scanning device, once an image is determined to have successfully spliced ​​with the previous image and successfully completed 3D reconstruction, the acquisition time of the image is recorded. Images acquired after the image and successfully completed 3D reconstruction are then determined to be valid images.

[0071] In some embodiments of the present disclosure, the detection of the current scanning environment type of the 3D scanning device based on the target image may be performed upon detecting that a user has triggered a designated interactive component. Upon triggering the designated interactive component, the image captured by the 3D scanning device begins to be used for 3D reconstruction. For example, a user may typically begin scanning a target object by clicking a "Start Scan" button on the scanning software. At this point, the scanning software begins acquiring images captured by the 3D scanning device and begins the 3D reconstruction process. Therefore, the current scanning environment type may be detected upon detecting that the user has clicked this button.

[0072] In some embodiments of the present disclosure, the timing for detecting the current scanning environment type of the three-dimensional scanning device based on the target image may be when it is detected that there is a large difference between the two frames of images captured before and after the three-dimensional scanning device is detected. Considering that the user may switch the scanning environment during the scanning process, for example, after the user clicks the "Start Scan" button on the scanning software, the patient's mouth is first scanned. After the scan is completed, the three-dimensional scanning device is moved out of the patient's mouth and the tooth model outside the mouth is scanned. In order to determine whether the scanning environment has changed and whether the scanning environment type detection step needs to be performed again, the two frames of images captured continuously by the three-dimensional scanning device can be continuously compared. When it is determined that the difference in the image content of the two consecutive frames exceeds a preset difference threshold, the step of determining the current scanning environment type of the three-dimensional scanning device based on the target image is executed.

[0073] In some embodiments of the present disclosure, before prompting a user to switch the current operating mode of the 3D scanning device to a target operating mode that matches the type of scanning environment, a determination may be made as to whether the current operating mode of the 3D scanning device is consistent with the target operating mode. If not, the user is prompted to switch the current operating mode of the 3D scanning device to the target operating mode that matches the type of scanning environment.

[0074] In some embodiments of the present disclosure, the method can be performed by a terminal device that is communicatively connected to the three-dimensional scanning device, and the three-dimensional scanning device is configured to send the captured image to the terminal device so that the terminal device can perform three-dimensional reconstruction of the target object based on the received image, wherein the terminal device includes a display interface configured to display the real-time reconstructed three-dimensional model. When prompting the user to switch the current operating mode of the three-dimensional scanning device to a target operating mode that matches the current scanning environment type, a prompt message can be displayed through the display interface to prompt the user to switch the current operating mode of the three-dimensional scanning device to the target operating mode. For example, the terminal device can be installed with scanning software, and the scanning software usually displays the real-time reconstructed three-dimensional model to the user through an interactive interface so that the user can understand the current scanning status. Therefore, the user can be prompted in the interactive interface through text, patterns, a combination of text and patterns, etc. to prompt the user to switch the current operating mode to the target operating mode.

[0075] In some embodiments of the present disclosure, the 3D scanning device may be an oral 3D scanning device, and the scanning environment types include intraoral and extraoral environments. When determining the current scanning environment type, a target image may be input into a pre-trained scanning environment detection model, which then predicts the current scanning environment type. The scanning environment detection model may be trained using a large number of sample images labeled with the scanning environment type.

[0076] In some embodiments of the present disclosure, taking into account that the extraoral environment is usually diverse, for example, in the scenario of scanning a tooth model outside the mouth, since the materials of the tooth model usually include many categories, such as metal, resin, paraffin, etc., and the sample image is difficult to cover all the scenes of the extraoral environment, therefore, it is difficult for the detection model trained using the sample image to accurately predict some scenes that are not covered by the sample image, resulting in poor accuracy of the detection results of the pre-trained scanning environment detection model. In order to improve the performance of the scanning environment detection model, in some embodiments of the present disclosure, a contrastive learning mechanism can be introduced when training the above-mentioned model, that is, contrastive loss can be introduced when determining the loss of the model. By introducing contrastive loss, the model can be constrained so that when the model extracts features from images, for images of the same category (that is, images of the same scanning environment type), the extracted features are as close as possible, and conversely, for images of different categories, the extracted features are as far apart as possible.

[0077] For example, in some embodiments of the present disclosure, when training a scanning environment detection model, at least two sample images may be obtained, where the at least two sample images correspond to the same scanning environment type. Feature extraction is then performed on the at least two sample images using a preset initial model to obtain features of the at least two sample images. A target loss may then be determined based on the difference between the similarity of features between each of the at least two sample images and a preset similarity threshold. The model parameters of the initial model are then adjusted based on the target loss to train the initial model and obtain the aforementioned scanning environment detection model. Generally, if the two sample images correspond to the same scanning environment type, the features of the two sample images should also be similar. Therefore, a similarity threshold may be pre-set. If the two sample images correspond to the same scanning environment type, the similarity of features between the two sample images should be very close to the similarity threshold. Therefore, the target loss may be determined based on the degree of similarity between the two features to train the initial model. This allows the trained model to extract features closer to those of images with the same scanning environment type.

[0078] In some embodiments of the present disclosure, when training the above-mentioned scanning environment detection model, sample image triplets may also be obtained, where each set of sample image triplets includes a first sample image, a second sample image having the same scanning environment type as the first sample image, and a third sample image having a different scanning environment type from the first sample image. A preset initial model may then be used to perform feature extraction on the first, second, and third sample images, respectively, to obtain their respective features. A target loss may then be determined based on the similarity between the features of the second sample image and the features of the first sample image, and the similarity between the features of the third sample image and the features of the first sample image, and the network parameters of the initial model may be adjusted based on the target loss to train the initial model. For two sample images with the same scanning environment type, the similarity between their features should be higher than the similarity between the features of two sample images with different scanning environment types. Determining the target loss based on this principle and adjusting the model parameters of the initial model can make the trained scanning environment detection model more accurate in extracting features from images, and thus, more accurate in detection results.

[0079] In some embodiments of the present disclosure, considering that the intraoral environments are often relatively similar and the differences between them are not large, the image features of the images collected in the intraoral environment are often relatively similar, that is, the feature vectors of the images will be distributed within a certain approximate range in the feature space. Therefore, for the detection of the intraoral environment, a large number of sample images collected by the oral three-dimensional scanning device in the intraoral environment can also be obtained in advance, and then feature extraction can be performed on the sample images to obtain the features of these sample images, and the cluster centers of the features of the sample images can be determined. For example, feature vectors representing the features of the sample images can be obtained, and the cluster centers of these feature vectors can be determined. Subsequently, when determining the current scanning environment type of the oral three-dimensional scanning device based on the target image, feature extraction can be performed on the target image to obtain the features of each frame of the target image, and then, based on the degree of proximity between the features of each target image and the feature cluster center, it can be determined whether the current scanning environment type of the oral three-dimensional scanning device is an intraoral environment. For example, if the feature of the target image is very close to the center of the feature cluster, it means that the scanning environment type corresponding to the target image is an intraoral environment, where the degree of closeness can be expressed by the distance between the two, for example, it can be expressed by Euclidean distance, Manhattan distance, etc. If the distance between the two is less than a preset distance threshold, it is considered that the two are close, that is, it is determined that the scanning environment type corresponding to the target image is an intraoral environment.

[0080] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space reasons, they are not listed one by one in the embodiments of this disclosure.

[0081] Accordingly, the embodiment of the present disclosure further provides a control device for a three-dimensional scanning device, as shown in FIG6 , the control device includes:

[0082] An acquisition module 61 is configured to acquire a target image captured by a three-dimensional scanning device;

[0083] A scanning environment type determination module 62 is configured to determine a current scanning environment type of the three-dimensional scanning device based on the target image;

[0084] The processing module 63 is configured to switch the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type, or prompt the user to switch the current working mode of the three-dimensional scanning device to the target working mode that matches the scanning environment type;

[0085] Among them, the three-dimensional scanning device is configured with at least two working modes, each working mode corresponds to a scanning environment type. When the three-dimensional scanning device is in different working modes, the image is processed differently during the process of three-dimensional reconstruction of the target object using the image captured by the three-dimensional scanning device; and / or the operation mode of each component in the three-dimensional scanning device is different during the process of capturing the image using the three-dimensional scanning device.

[0086] The specific steps of the control method for executing the three-dimensional scanning device by the control device can be referred to the description in the above method embodiment, which will not be repeated here.

[0087] Furthermore, an embodiment of the present disclosure also provides a terminal device, as shown in Figure 7, the terminal device includes a processor 71, a memory 72, and computer instructions stored in the memory 72 for execution by the processor 71. When the processor 71 executes the computer instructions, it implements the control method of the three-dimensional scanning device of any one of the above embodiments.

[0088] Among them, the specific steps of the above-mentioned terminal device executing the control method of the three-dimensional scanning device can refer to the description in the above-mentioned method embodiment, and will not be repeated here.

[0089] Furthermore, an embodiment of the present disclosure also provides a control system for a three-dimensional scanning device, which includes the terminal device mentioned in the above embodiment and a three-dimensional scanning device communicatively connected to the terminal device. The three-dimensional scanning device is configured to capture images of the area to be scanned and send the images to the terminal device.

[0090] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method of the three-dimensional scanning device of any of the aforementioned embodiments.

[0091] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette storage, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be configured to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0092] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment of the embodiments of the present disclosure or certain parts of the embodiments.

[0093] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0094] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, and the modules described as separate components may or may not be physically separated. When implementing the embodiment of the present disclosure, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0095] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the embodiment of the present disclosure. Industrial Applicability

[0096] The technical solution provided by the present disclosure can be applied to the field of 3D scanning technology. The technical solution of the present disclosure can automatically determine the current scanning environment type of the 3D scanning device based on the images collected by the 3D scanning device, and then automatically switch the working mode of the 3D scanning device to a target working mode that is adapted to the scanning environment type, or prompt the user to switch the working mode of the 3D scanning device to a target working mode that is adapted to the scanning environment type, thereby avoiding the problem that the user forgets to set the working mode, resulting in poor effect of the final reconstructed 3D model, and the need for re-3D reconstruction or re-scanning.

Claims

1. A control method for a three-dimensional scanning device, comprising: Acquire a target image acquired by a three-dimensional scanning device; Determining a current scanning environment type of the three-dimensional scanning device based on the target image; Switching the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type, or prompting the user to switch the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type; In which, the three-dimensional scanning device is configured with at least two working modes, each working mode corresponds to a scanning environment type. When the three-dimensional scanning device is in different working modes, in the process of three-dimensionally reconstructing the target object using the image captured by the three-dimensional scanning device, the image is processed in a different way, and / or, in the process of capturing the image using the three-dimensional scanning device, the operation mode of each component in the three-dimensional scanning device is different.

2. The method according to claim 1, wherein: The different processing methods include: whether the processing steps for processing the image include a target step; wherein the target step is set to determine data in the image that is irrelevant to the three-dimensional reconstruction of the target object to be reconstructed, so as to ignore the data when generating the three-dimensional model of the target object.

3. The method according to claim 2, wherein: The three-dimensional scanning device is an oral three-dimensional scanning device, the scanning environment types include an intraoral environment and an extraoral environment, and the working mode includes an intraoral working mode corresponding to the intraoral environment and an extraoral working mode corresponding to the extraoral environment; When the working mode is the intraoral working mode, the processing step of processing the image includes the target step; When the working mode is the extraoral working mode, the processing steps for processing the image do not include the target step.

4. The method according to claim 1, wherein: The different processing methods include: different processing parameters when processing the image, and the processing parameters include one or more of the following: The enhancement amplitude when the brightness of the image is enhanced, and the maximum stitching error allowed when the images are stitched.

5. The method according to claim 4, wherein: The three-dimensional scanning device is an oral three-dimensional scanning device, the scanning environment types include an intraoral environment and an extraoral environment, and the working mode includes an intraoral working mode corresponding to the intraoral environment and an extraoral working mode corresponding to the extraoral environment; When the working mode is the intra-oral working mode, the enhancement amplitude is the first amplitude, and the maximum stitching error is the first stitching error; When the working mode is the extraoral working mode, the enhancement amplitude is the second amplitude, and the maximum stitching error is the second stitching error; The first amplitude is greater than the second amplitude, and the first splicing error is greater than the second splicing error.

6. The method according to claim 3 or 5, wherein: The intra-oral working mode includes at least two sub-modes, and the tooth loss conditions in the scanned oral cavity are different in different sub-modes, and the processing methods for processing the image in different sub-modes are different; and / or The extraoral working mode includes a plurality of sub-modes. The materials of the scanned tooth models in different sub-modes are different, and the processing methods for processing the images in different sub-modes are different.

7. The method according to claim 1, wherein: The target image includes multiple frames, and the acquisition time interval between two adjacent target images in the multiple target images exceeds a preset time length. The determining the current scanning environment type of the three-dimensional scanning device based on the target image includes: Detecting the current scanning environment type of the three-dimensional scanning device based on each target image in the multiple target image frames to obtain a detection result; if the detection results corresponding to the multiple target image frames are consistent, using the current detection result as the current scanning environment type of the three-dimensional scanning device; and / or The target image is a valid image selected from the images acquired by the three-dimensional scanning device, and the valid image is obtained based on the following method: after successfully realizing three-dimensional reconstruction using the images acquired by the three-dimensional scanning device, if the current frame image is successfully spliced ​​with the frame image before the current frame image, the acquisition time of the current frame image is recorded; and the image whose acquisition time is after the acquisition time of the current frame image and which successfully realizes three-dimensional reconstruction is determined as a valid image.

8. The method according to claim 1, wherein: The determining, based on the target image, the current scanning environment type of the three-dimensional scanning device comprises: In the case of detecting that a user triggers a designated interactive component, determining the current scanning environment type of the three-dimensional scanning device based on the target image, wherein after the designated interactive component is triggered, the image collected by the three-dimensional scanning device begins to be used for three-dimensional reconstruction; or When it is determined that the difference in picture content between two frames of images continuously acquired by the three-dimensional scanning device exceeds a preset difference threshold, the current scanning environment type of the three-dimensional scanning device is determined based on the target image.

9. The method according to claim 1, wherein: The prompting the user to switch the current working mode of the three-dimensional scanning device to a target working mode matching the scanning environment type includes: Determining whether the current working mode of the three-dimensional scanning device is consistent with the target working mode; If they are inconsistent, the user is prompted to switch the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type.

10. The method according to claim 9, wherein: The method is performed by a terminal device that is communicatively connected to the three-dimensional scanning device, wherein the three-dimensional scanning device is configured to send the acquired image to the terminal device so that the terminal device performs three-dimensional reconstruction of the target object based on the received image; the terminal device includes a display interface configured to display the three-dimensional model reconstructed in real time; The prompting the user to switch the current working mode of the three-dimensional scanning device to a target working mode matching the scanning environment type includes: Prompt information is displayed through the display interface to prompt the user to switch the current working mode of the three-dimensional scanning device to a target working mode that matches the scanning environment type.

11. The method according to claim 1, wherein: The three-dimensional scanning device includes an oral three-dimensional scanning device, the scanning environment type includes an intraoral environment and an extraoral environment, the scanning environment type is determined by detecting the target image through a pre-trained scanning environment detection model, and the scanning environment detection model is trained in the following manner: Acquire at least two frames of sample images, the scanning environment types corresponding to the at least two frames of sample images are the same; use a preset initial model to extract features from the at least two frames of sample images respectively to obtain features of the at least two frames of sample images; determine a target loss based on the difference between the similarity of features of each pair of sample images in the at least two frames of sample images and a preset similarity threshold, and adjust the model parameters of the initial model based on the target loss to train and obtain the scanning environment detection model; or Acquire sample image triplets, each group of sample image triplets includes a first sample image, a second sample image having the same scanning environment type as the first sample image, and a third sample image having a different scanning environment type from the first sample image; use a preset initial model to perform feature extraction on the first sample image, the second sample image, and the third sample image to obtain respective features; determine a target loss based on the similarity between the features of the second sample image and the features of the first sample image, and the similarity between the features of the third sample image and the features of the first sample image, and adjust the model parameters of the initial model based on the target loss to train a scanning environment detection model.

12. The method according to claim 1, wherein: The three-dimensional scanning device includes an oral three-dimensional scanning device, the scanning environment type includes an intraoral environment and an extraoral environment, and determining the current scanning environment type of the three-dimensional scanning device based on the target image includes: Performing feature extraction on the target image to obtain features of the target image; Based on the degree of proximity between the features of the target image and a preset feature clustering center, determine whether the current scanning environment type of the oral 3D scanning device is an intraoral environment; wherein the feature clustering center is the clustering center of the features of multiple frames of sample images, and the sample images are images collected by the oral 3D scanning device in the intraoral environment.

13. A control device for a three-dimensional scanning device, comprising: An acquisition module, configured to acquire a target image acquired by a three-dimensional scanning device; A scanning environment type determination module, configured to determine a current scanning environment type of the three-dimensional scanning device based on the target image; a processing module, configured to switch the current working mode of the three-dimensional scanning device to a target working mode matching the scanning environment type, or to prompt a user to switch the current working mode of the three-dimensional scanning device to a target working mode matching the scanning environment type; In which, the three-dimensional scanning device is configured with at least two working modes, each working mode corresponds to a scanning environment type. When the three-dimensional scanning device is in different working modes, in the process of three-dimensionally reconstructing the target object using the image captured by the three-dimensional scanning device, the image is processed in a different way, and / or, in the process of capturing the image using the three-dimensional scanning device, the operation mode of each component in the three-dimensional scanning device is different.

14. A terminal device, comprising: A processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein the processor implements the method as described in any one of claims 1-12 when executing the computer instructions.

15. A control system for a three-dimensional scanning device, comprising: The terminal device as claimed in claim 14, and a three-dimensional scanning device communicatively connected to the terminal device, wherein the three-dimensional scanning device is configured to capture an image of the area to be scanned and send the image to the terminal device.

16. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images