Automated intraoral scanning robot system

Abstract

According to one embodiment of the present disclosure, an oral scanning system is disclosed. The oral scanning system may include: a multi-axis robotic arm; an oral scanner coupled to the end of the multi-axis robotic arm to acquire three-dimensional data of a scan target; and a control unit that generates and controls the movement of the robotic arm and creates a three-dimensional model by aligning the three-dimensional data acquired from the oral scanner based on the pose of the oral scanner calculated from the kinematic information of the robotic arm.

Description

Technology Field

[0001] The present invention relates to an automatic oral scanning system and a method thereof. More specifically, it relates to a system that automatically generates a three-dimensional model of teeth and surrounding tissues using an oral scanner mounted on a multi-axis robotic arm and an external third-person view camera, and a control method thereof. Background Technology

[0002] Recently, the introduction of digital technology in the field of dental care has been active, and in particular, 3D digital impression taking using oral scanners is widely used. An oral scanner is a device that is held by a skilled operator, such as a doctor or dental technician, to scan the inside of a patient's oral cavity and generate a 3D model of the teeth and surrounding tissues.

[0003] To generate such a 3D model, most oral scanners operate based on Simultaneous Localization and Mapping (SLAM) technology, which estimates the position and pose of the camera from continuous color and depth (RGBD) images and simultaneously creates a 3D map.

[0004] However, conventional SLAM-based oral scanning methods had the following problems.

[0005] First, since the entire scanning process depends entirely on the operator's skill level, it is difficult to guarantee the consistency and accuracy of the scan results. Additionally, operator fatigue increases during prolonged scanning, which may lead to a deterioration in scan quality.

[0006] Second, SLAM technology has a chronic problem of being vulnerable to drift. When the scan path becomes long or when capturing areas lacking feature points, such as tooth surfaces, errors gradually accumulate in the estimated camera pose. These accumulated errors cause alignment errors and severe distortion in the final 3D model.

[0007] Third, to correct accumulated errors, Loop Closure Detection, which returns to the scan start point to form a loop, or optimizing the pose of the entire scan path

[0008] Complex and computationally intensive post-processing steps, such as pose graph optimization and bundle adjustment, are essential. These processes not only delay the overall scanning time but also fail to guarantee perfect error correction.

[0009] Therefore, there was a demand for a new technology that could generate a 3D oral model quickly and accurately by fundamentally solving the problem of cumulative error in SLAM technology while minimizing operator intervention.

[0010] Republic of Korea Published Patent: 10-2015-0138784 The problem to be solved

[0011] The present disclosure is conceived based on the aforementioned background technology and aims to eliminate result variations due to operator skill level and always provide standardized, high-quality scan data by automating the oral scanning process using a robotic arm.

[0012] The present disclosure aims to drastically improve the accuracy of a 3D oral model by fundamentally solving the chronic problem of drift in SLAM through the direct use of precise kinematic pose information of a robotic arm rather than SLAM technology when generating a 3D model.

[0013] The present disclosure aims to significantly reduce the need for complex post-processing correction algorithms, such as loop closure detection, thereby shortening the overall scanning and data processing speed.

[0014] The present disclosure aims to provide a fully automated oral scanning system that autonomously generates and executes a scan path optimized for the patient's oral structure through a deep learning-based intelligent policy model. means of solving the problem

[0015] According to one embodiment of the present disclosure for realizing the aforementioned objectives, an oral scanning system is disclosed. The oral scanning system may include: a multi-axis robotic arm; an oral scanner coupled to the end of the multi-axis robotic arm to acquire three-dimensional data of a scan target; and a control unit that generates and controls the movement of the robotic arm and creates a three-dimensional model by aligning the three-dimensional data acquired from the oral scanner based on the pose of the oral scanner calculated from the kinematic information of the robotic arm.

[0016] Alternatively, the system may further include a third-person view camera positioned to photograph the robot arm and the scan target, and the control unit may generate a motion of the robot arm for the oral scanner to enter the oral cavity of the scan target based on an image acquired from the third-person view camera.

[0017] Alternatively, the third-person view camera mentioned above may be an RGBD camera that acquires both color images and depth images.

[0018] Alternatively, the control unit may perform initialization to establish the global coordinate system of the scan target by recognizing the face of the scan target through the third-person view camera at the start of the scanning process.

[0019] Alternatively, the global coordinate system may be fixed to a specific anatomical feature of the scanned object recognized by the third-person view camera, independently of the base coordinate system of the robot arm.

[0020] Alternatively, the control unit can calculate the pose of the oral scanner by performing a forward kinematic operation based on the kinematic information including the joint angles of each of the plurality of joints of the multi-axis robot arm.

[0021] Alternatively, the pose of the oral scanner can be finally calculated using a pre-calibrated Hand-Eye transformation matrix that defines the relationship between the transformation matrix of the end coordinate system with respect to the base coordinate system of the multi-axis robot arm and the sensor coordinate system of the oral scanner with respect to the end coordinate system.

[0022] Alternatively, the control unit may finally generate the three-dimensional model without performing loop closure detection or pose graph optimization processes for the entire scan path of the oral scanner.

[0023] Alternatively, the above 3D model may be a 3D point cloud or a voxel grid accumulated on a global coordinate system.

[0024] Alternatively, it may further include an emergency stop means that can be activated by an operator or a scan object and immediately stops all movements of the multi-axis robot arm upon activation.

[0025] Alternatively, the control unit may include a deep learning-based policy model that determines the next action of the robot arm based on data received from the oral scanner.

[0026] Alternatively, the control unit may include a GPU to accelerate the computation of the deep learning-based policy model.

[0027] Alternatively, the policy model can generate the motion of the robot arm by processing multi-modal inputs including images from the third-person view camera, images from the oral scanner, and 3D model data restored to date.

[0028] Alternatively, the policy model may receive real-time images acquired from the oral scanner and a 3D model restored to date as input, and output an operation including joint angle values ​​for each of the multiple joints of the multi-axis robot arm.

[0029] Alternatively, the action output by the policy model may be a delta pose including relative displacement and rotation with respect to the current pose of the oral scanner.

[0030] Alternatively, the policy model may include a first policy that generates an oral entry action based on images from a third-person view camera; and a second policy that generates an oral internal scanning action based on data from the oral scanner and a three-dimensional model generated so far.

[0031] Alternatively, the first policy may perform the oral entry operation by generating a path that avoids collision between the face of the scanned object and the oral scanner using depth information acquired from the third-person view camera.

[0032] Alternatively, the second policy can generate an internal scanning operation by receiving the data from the oral scanner and the image acquired from the third-person view camera together.

[0033] Alternatively, the second policy may adjust the operation of the robot arm by detecting the risk of collision with soft tissue or the movement of the scan target using images from the third-person view camera during the internal scanning operation.

[0034] Alternatively, the policy model may be a single integrated model that receives data from the third-person view camera and the oral scanner and sequentially generates oral entry actions and internal scanning actions.

[0035] Alternatively, the policy model may determine whether the generated 3D model satisfies a pre-set quality standard to determine the termination of the process, and when the termination is determined, generate an action to move the oral scanner out of the oral cavity.

[0036] Alternatively, the policy model may identify hole areas in the generated 3D model where data is missing or density is lower than a threshold before deciding to terminate the scanning process, and generate additional scan paths to fill the identified hole areas.

[0037] Alternatively, the above policy model can be learned through reinforcement learning or imitation learning.

[0038] Alternatively, the training of the above policy model can be performed using an offline training method utilizing a pre-built expert demonstration dataset or a scanning dataset.

[0039] Alternatively, the training of the above policy model can be performed by applying data augmentation techniques to increase the volume and diversity of the collected training data.

[0040] Alternatively, the aforementioned imitation learning can utilize a diffusion policy model that models expert behavior as a probability distribution and generates behavior from noise.

[0041] Alternatively, the above imitation learning can be learned using a dataset collected by synchronizing third-person view camera footage at the time of the operation, oral scanner footage, and joint angle data of the robot arm operated by the expert from a scanning motion demonstrated by an expert through remote operation.

[0042] Alternatively, the expert data can be collected through direct teaching in which an expert directly grasps and moves the multi-axis robot arm with their hand to demonstrate the scanning path.

[0043] Alternatively, the control unit may generate the expert data by synchronizing data streams output from the third-person view camera, the oral scanner, and the joint encoder of the multi-axis robotic arm based on timestamps.

[0044] Alternatively, the reinforcement learning above can train the policy model using a reward function designed to maximize the range and data quality of the scanned tooth region.

[0045] Alternatively, the compensation function may be designed to provide a positive compensation proportional to the amount of valid data newly added to the 3D model after the unit operation of the robot arm; and a negative compensation when scanning an existing scanned area in duplicate.

[0046] Alternatively, the compensation can be calculated by updating the 3D model using an accurate pose calculated from the kinematic information of the robot arm, and then quantitatively measuring the difference between the model before the update and the model after the update.

[0047] Alternatively, the above three-dimensional model may be generated to have a pre-set spatial resolution to satisfy the precision required for fabricating dental prostheses or surgical guides.

[0048] Alternatively, each 3D point or voxel of the above 3D model may be a texture-mapped model that further includes color (RGB) information based on a color image acquired from the oral scanner.

[0049] Alternatively, the control unit may apply a surface reconstruction algorithm to the accumulated three-dimensional point cloud or voxel grid to finally generate a three-dimensional model in the form of a polygon mesh.

[0050] Alternatively, an automatic oral scanning method is disclosed, which is performed by an oral scanning system comprising a multi-axis robotic arm and an oral scanner coupled to the end of the robotic arm. The method may include the steps of: generating and controlling the motion of the robotic arm; calculating a pose of the oral scanner from kinematic information of the robotic arm; and creating a three-dimensional model by aligning three-dimensional data acquired from the oral scanner based on the calculated pose.

[0051] Alternatively, a computer program stored on a computer-readable storage medium is disclosed. When executed on a computer device, the computer program performs the following methods for automatic oral scanning and may include: a step of generating and controlling the movement of a robot arm; a step of calculating a pose of an oral scanner from kinematic information of the robot arm; and a step of creating a three-dimensional model by aligning three-dimensional data acquired from the oral scanner based on the calculated pose. Effects of the invention

[0052] The present disclosure fundamentally solves the problem of cumulative error in SLAM by utilizing precise kinematic pose information of a robotic arm. Through this, it is possible to generate a highly precise 3D oral model without distortion, which can dramatically improve the accuracy of fabricating prosthetics and surgical guides.

[0053] The present disclosure reduces or eliminates complex loop closure detection and pose graph optimization calculations. Consequently, the total processing time from scan completion to verification of the final model is shortened, thereby increasing clinical efficiency.

[0054] The present disclosure enables the reproducible acquisition of consistently high-quality scan results through an automated robotic system, as the scan results are not affected by the operator's skill level. This enables the standardization of dental treatment.

[0055] The present disclosure allows medical staff to be freed from time-consuming and fatigue-inducing scanning tasks. This enables them to focus on more important tasks, such as patient care and treatment planning. Brief explanation of the drawing

[0056] FIG. 1 is a diagram showing the overall configuration of an automatic oral scanning system according to one embodiment of the present invention. FIG. 2 is a block diagram showing the data flow of a control unit and a policy model according to an embodiment of the present invention. FIG. 3 is a flowchart showing the entire process of an automatic oral scanning method according to one embodiment of the present invention. FIG. 4 illustrates a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Specific details for implementing the invention

[0057] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0058] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computer device and the computer device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0059] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0060] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0061] And, the term “at least one of A or B” should be interpreted to mean “a case including only A,” “a case including only B,” or “a combination of A and B.”

[0062] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0063] The description of the presented embodiments is provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0065] FIG. 1 is a diagram showing the overall configuration of an automatic oral scanning system according to one embodiment of the present invention.

[0066] Referring to FIG. 1, the automatic oral scanning system of the present invention comprises a unit chair (20) on which a patient (10), who is the subject of the scan, is positioned, an oral scanner (200) for scanning the inside of the oral cavity of the patient (10), a multi-axis robotic arm (300) for grasping and precisely moving the oral scanner (200), a third-person view camera (120) for capturing the facial portion of the patient (10) and the operation of the multi-axis robotic arm (300), and a control unit (100) for controlling the operation of these components and processing data.

[0067] The patient (10) sits or lies down in a unit chair (20) to undergo a scanning procedure. The unit chair (20) may be a chair used in a general dental practice environment and may include a function to fix or adjust the patient's (10) position to a location optimized for scanning.

[0068] The multi-axis robot arm (300) is installed with its base fixed near or integrally with the unit chair (20). The multi-axis robot arm (300) is implemented, for example, as a vertical multi-joint robot having six or more degrees of freedom, and performs the role of freely controlling the position and orientation of the scanner within a complex and narrow oral cavity.

[0069] An oral scanner (200) is coupled to the end of the multi-axis robot arm (300), that is, the end-effector. The oral scanner (200) acquires three-dimensional data containing shape information of the teeth and surrounding soft tissues through a sensor unit located at the end. For example, three-dimensional point cloud data can be generated by continuously capturing RGBD images including color (RGB) images and depth images.

[0070] A third-person view camera (120) is installed outside the system to secure a wide field of view that includes the patient's (10) face, a multi-axis robotic arm (300), and an oral scanner (200). The third-person view camera (120) performs the role of recognizing the initial position for the oral scanner (200) to safely enter the patient's (10) oral cavity, or detecting the patient's (10) movements or changes in the external environment that may occur during scanning. In one embodiment, the third-person view camera (120) may be implemented as an RGBD camera capable of acquiring depth information.

[0071] The control unit (100) is connected to the multi-axis robot arm (300), oral scanner (200), and third-person view camera (120) via wired or wireless connection to transmit and receive data and acts as the brain of the system that controls them. The control unit (100) can typically be implemented as a computer system equipped with a GPU.

[0072] Specifically, the control unit (100) receives image or 3D data from a third-person view camera (120) and an oral scanner (200), determines the next action through a built-in intelligent policy model (not shown), and transmits a control signal to a multi-axis robot arm (300).

[0073] In particular, according to the core features of the present invention, the control unit (100) does not use the pose estimated by the SLAM function of the oral scanner (200) itself when generating the final 3D oral model. Instead, it uses kinematic information such as joint angle values ​​measured from encoders attached to each joint of the multi-axis robot arm (300). The control unit (100) calculates the mathematically accurate position and orientation of the oral scanner (200) through this kinematic information and a forward kinematics model, and aligns the 3D data acquired by the oral scanner (200) with the global coordinate system based on the calculated pose to generate a precise 3D model without cumulative error.

[0075] FIG. 2 is a block diagram showing the data flow of a control unit and a policy model according to an embodiment of the present invention.

[0076] Referring to FIG. 2, the control unit (100), which is a core component of the present invention, includes an intelligent policy model (110) internally and controls the entire system by organically linking with a three-dimensional model generator (400).

[0077] First, the policy model (110) receives data from various sources to autonomously generate a scanning path. A third-person view camera (120) monitoring the external environment of the system provides the policy model (110) with image information of an external view, such as the patient's face, the position of the mouth, and the posture of the robot arm. An oral scanner (200) mounted on the end of the robot arm provides the policy model (110) with real-time images and 3D scan data of the inside of the oral cavity. Additionally, the restoration status of the 3D model generated so far (e.g., information on which parts were scanned and which parts were missing) is fed back to the policy model (110) from the 3D model generator (400).

[0078] The policy model (110) comprehensively analyzes the multi-modal data input as described above and determines the optimal action that the multi-axis robot arm (300) must perform for the next scan. The determined action is transmitted to the multi-axis robot arm (300) in the form of a robot arm control signal so that the robot arm moves precisely.

[0079] At the same time, the control unit (100) calculates the global coordinate system-based precise pose information (410) of the oral scanner (200) in real time based on the robot arm control signal or kinematic information obtained from the joint encoder of the multi-axis robot arm (300). This precise pose information (410) is a mathematically accurate position and attitude value that does not include the cumulative error problem of SLAM.

[0080] The 3D model generator (400) receives two types of data to generate a 3D model. One is the original 3D scan data (e.g., point cloud) of each frame received directly from the oral scanner (200), and the other is the precise pose information (410) at the time the frame was captured, received from the control unit (100).

[0081] The 3D model generator (400) accurately aligns and accumulates 3D scan data in a global coordinate system based on the precision pose information (410). Through this process, a high-precision final 3D model (420) is completed in which distortion and accumulation errors are fundamentally eliminated.

[0083] FIG. 3 is a flowchart showing the entire process of an automatic oral scanning method according to an embodiment of the present invention. The automatic oral scanning method of the present invention is largely composed of an initialization step (500), an entry step (600), a scanning loop (700), and an exit step (800).

[0084] Initialization phase (500)

[0085] First, an initialization step (500) is performed to start the scanning process. In the patient position setting (510) step, the patient subject to the scan assumes a position for scanning, such as sitting in a unit chair. Next, a step of recognizing the face with a third-person camera (520) is performed. The control unit captures the entire or part of the patient's face through the third-person camera and recognizes facial feature points. Based on the recognized feature points, the control unit establishes a system global coordinate system (530) fixed to the patient's position. This global coordinate system subsequently serves as the reference for all robot arm movements and 3D data alignment.

[0086] Entry Phase (600)

[0087] Once initialization is complete, an entry step (600) is performed to safely move the oral scanner into the patient's oral cavity. The control unit identifies the 3D position of the oral cavity using a third-person camera (610). Using depth information from an RGBD camera, the exact 3D spatial position and degree of opening of the oral cavity, such as the patient's lips and teeth, are identified. Subsequently, the entry step (620) of the learned policy model is activated. The control unit inputs the identified oral position information into the policy model, and the policy model generates an optimal path for safe entry without collision. According to the generated path, the control unit moves the scanner into the oral cavity (630) using a robotic arm.

[0088] Scanning loop (700)

[0089] When the oral scanner reaches the scan start position in the mouth, it repeatedly performs a scanning loop (700) to scan the entire dentition.

[0090] First, the control unit activates the scanning phase of the policy model (710). In this phase, the policy model receives real-time images and 3D reconstruction results up to that point (720). That is, it receives the images of the inside of the oral cavity currently being captured by the oral scanner and the completeness information of the current 3D model being fed back from the 3D model generator.

[0091] Based on the input information, the policy model determines the next scan area, such as an area that has not yet been scanned or has low data quality, and generates a motion signal to control the robot arm (730).

[0092] While the robot arm moves under the control of the policy model, the system simultaneously performs data acquisition and 3D model generation. That is, the oral scanner follows the movement path and captures a 3D point cloud (750). At the same time, the control unit measures the joint angles from each joint encoder of the robot arm and calculates the forward kinematics (740) to calculate the precise pose of the oral scanner at the moment the 3D point cloud is captured.

[0093] Afterward, the control unit accumulates and aligns (760) the captured 3D point cloud in a global coordinate system based on the calculated precision pose. This gradually completes the 3D oral model.

[0094] Whenever a loop is completed, the control unit determines (770) whether the scan of all dentitions is complete. If the scan is not complete (no), it returns to step 720 and repeats the process of determining the next scan area.

[0095] Termination phase (800)

[0096] When it is determined that all teeth have been scanned with sufficient quality in the scanning loop (700) (e.g.), the system enters the termination phase (800). The control unit creates a safe exit path through the policy model and moves the scanner out of the oral cavity (810). Finally, the final 3D model is saved and the entire scanning process is terminated (820).

[0098] The main components of the present invention, their combination relationships, and operating principles have been explained through the aforementioned FIGS. 1 to 3. Below, the components of the present invention will be explained in detail once again based on the contents of each claim.

[0099] The multi-axis robot arm (300) of the present invention is a component that is equipped with an oral scanner (200) at its end and moves precisely in three-dimensional space according to a control signal from a control unit (100) to perform a scanning operation.

[0100] The above-mentioned multi-axis robot arm (300) is preferably implemented as a vertical multi-joint robot having six or more degrees of freedom to freely control the position and orientation of the oral scanner (200) within the narrow and complex oral cavity of the patient (10). Through this, not only the three-dimensional position (X, Y, Z) of the scanner but also the three-dimensional rotational orientation (Roll, Pitch, Yaw) can be precisely controlled. In particular, considering the characteristics of a medical environment where operation must be performed at a close distance to the patient, it is preferable to use a collaborative robot with built-in safety functions such as collision detection.

[0101] Above all, to achieve the purpose of the present invention, the multi-axis robot arm (300) must have high precision and repeatability. Each joint is equipped with a high-resolution encoder capable of measuring the rotation angle of the robot with great precision. The control unit (100) obtains kinematic information including the angle values ​​of each joint measured in real time from this encoder. This information serves as the basis for calculating the precision pose information (410) that the present invention uses to eliminate the cumulative error of SLAM.

[0102] Examples of multi-axis robot arms (300) that can be used in the present invention include the UR series of Universal Robots and the Piper of AgileX, but are not limited thereto.

[0103] The oral scanner (200) is a sensor device that is coupled to the end of a multi-axis robot arm (300) to directly acquire three-dimensional shape data of a scan target such as the patient's (10) teeth and surrounding tissues.

[0104] The oral scanner (200) of the present invention may use a hand-held type oral scanner that is generally available on the market. For example, a scanner that simultaneously acquires depth information and color (RGB) information to the surface of an object using a confocal microscopy method or a structured light method may be used. The scanner generates three-dimensional point cloud data containing color information for each captured frame and transmits it to the control unit (100). Examples of oral scanners (200) that can be used in the present invention include TRIOS from 3Shape and i900 from Medit, but any device capable of acquiring three-dimensional data is not limited to these and can be applied.

[0105] In particular, the oral scanner (200) in the present invention has a fundamental difference from the conventional method of use. In the prior art, the oral scanner generates a 3D model by estimating its own position and pose through a built-in SLAM algorithm. However, in the present invention, the pose information estimated by the oral scanner (200) itself through SLAM is not used or is discarded. Instead, the control unit (100) receives only pure 3D scan data (raw data) from the oral scanner (200), and the pose information for matching this data uses only values ​​calculated from the kinematic information of the multi-axis robot arm (300).

[0106] Additionally, the data acquired by the oral scanner (200) is utilized in two ways. First, the 3D point cloud data is transmitted to the 3D model generator (400) and used to form the final 3D model (420). Second, the 2D or 3D video stream captured in real time is provided as input to the policy model (110) and is used to determine the current scan status and the next action of the robot arm.

[0107] The control unit (100) is a central processing unit that serves as the brain of the automatic oral scanning system according to the present invention. The control unit (100) is connected to all components of the system, such as a multi-axis robotic arm (300), an oral scanner (200), and a third-person view camera (120), and processes data and generates control signals to oversee the overall operation of the system. Physically, the control unit (100) may be implemented as a high-performance computer equipped with a Graphics Processing Unit (GPU) to efficiently process the computation of a deep learning model.

[0108] The functions of the control unit (100) can be broadly divided into two categories: generating and controlling robot arm motion and generating a kinematic-based 3D model.

[0109] Robot arm motion generation and control

[0110] The control unit (100) intelligently generates and controls the movement of the multi-axis robot arm (300) through a policy model (110) mounted internally. The policy model (110) is a deep learning model trained in advance through reinforcement learning or imitation learning.

[0111] In the motion generation process, the policy model (110) receives external environment information from a third-person view camera (120), internal oral image information from an oral scanner (200), and state information of the 3D model restored so far as multimodal inputs. Based on these inputs, the policy model (110) outputs an optimal motion for the next scan, for example, a target joint angle value of the robot arm or a relative amount of movement (delta pose) from the current pose. The control unit (100) converts the motion generated by the policy model into a control signal that the robot arm can perform and transmits it to the multi-axis robot arm (300).

[0112] kinematics-based 3D model generation

[0113] The creation of a three-dimensional model is the part where the core technical features of the present invention are implemented. The control unit (100) can exclude conventional SLAM-based pose estimation in this process.

[0114] First, the control unit (100) receives kinematic information containing accurate joint angle values ​​in real time from each joint encoder of the multi-axis robot arm (300).

[0115] Second, the control unit (100) calculates the position and orientation of the robot arm end by applying forward kinematics operations to the received kinematic information. By additionally applying a pre-corrected hand-eye transformation matrix, precise pose information (410) of the oral scanner (200) sensor is finally obtained.

[0116] Third, the control unit (100) aligns and accumulates the original 3D scan data (point cloud) received from the oral scanner (200) in a global coordinate system using the precise pose information (410) calculated in the second step. This process is performed by a 3D model generator (400) module, and a final 3D model (420) is completed through iterative accumulation.

[0117] In addition to this, the control unit (100) is responsible for all overall functions necessary for system operation, such as system initialization, determining the termination condition of the scanning process, providing a graphical user interface (GUI) for the operator, and processing emergency stop signals.

[0118] The automatic oral scanning system of the present invention may further include a third-person view camera (120) to automate the entry into the oral cavity, which is the starting step of the scanning process, and to ensure safety.

[0119] A third-person view camera (120) is fixed outside the system and positioned to capture a wide field of view including the multi-axis robotic arm (300) and the patient's (10) face. This camera serves to enhance situational awareness by providing the system with an objective external view, which is different from the view inside the oral cavity provided by the oral scanner (200). Preferably, the third-person view camera (120) is implemented as an RGBD camera capable of acquiring three-dimensional depth information along with color (RGB) images. The depth information provides key information for determining the accurate three-dimensional position of the patient's face and oral cavity.

[0120] Oral entry motion generation process

[0121] The control unit (100) uses the image acquired from the third-person view camera (120) to generate the following motion of the robot arm for the oral scanner (200) to safely enter the oral cavity of the patient (10).

[0122] Oral position recognition: When the scanning process starts, the control unit (100) receives a real-time video stream from a third-person view camera (120). The control unit (100) applies computer vision and deep learning-based object recognition technology to this video to determine the exact three-dimensional spatial position and size of the oral cavity formed by the patient's facial contours, nose, chin, and lips.

[0123] Entry path generation: The identified three-dimensional position information of the oral cavity is input into the entry stage (or first policy) of the policy model (110) within the control unit (100). The policy model (110) generates a safe and efficient movement path from the current position of the robot arm to the target point, which is the scan start point inside the oral cavity, without colliding with other facial parts of the patient (nose, jaw, cheek, etc.).

[0124] Robot arm control: The control unit (100) transmits a control signal to move the multi-axis robot arm (300) along the path generated by the policy model (110). Through this, the oral scanner (200) is automatically positioned accurately inside the patient's oral cavity.

[0125] In this way, by using a third-person view camera (120), the first step of scanning can be safely and consistently automated without manual intervention by the operator.

[0126] In order to further enhance the function of the third-person viewpoint camera (120) in the present invention, it is preferable to use an RGBD camera that acquires both a color (RGB) image and a depth (Depth) image.

[0127] An RGBD camera is a device that integrates a depth sensor, which measures the distance from the camera to each pixel within a scene, in addition to the functions of a standard color camera. Depth measurement methods may include Structured Light, which projects a specific pattern of light and analyzes its distortion, or Time-of-Flight (ToF), which measures the time it takes for light to reflect off an object and return.

[0128] By using such an RGBD camera, the control unit (100) obtains a color point cloud containing three-dimensional spatial information, rather than a simple two-dimensional color image, from a third-person view camera (120) in real time.

[0129] The control unit (100) can use depth information from an RGBD camera to precisely calculate the accurate 3D coordinates, size, and direction of the oral cavity entrance formed by the patient's lips in millimeters. This is impossible with only 2D images and provides essential information for the robot arm to reach the target starting point inside the oral cavity without error.

[0130] The control unit (100) can model the three-dimensional shape of the patient's entire face, such as the nose, chin, and cheeks, in real time using depth information. Based on this, the policy model (110) can generate a safe path that fundamentally prevents collisions with the patient while the oral scanner (200) is moving. This significantly improves the safety and reliability of the system.

[0131] The automatic oral scanning system of the present invention performs an initialization process using a third-person view camera (120) at the start of scanning to set a global coordinate system that serves as a reference for the entire scanning process.

[0132] Coordinate system setup process

[0133] Facial recognition: When the patient (10) is positioned in the unit chair (20), the control unit (100) captures the patient's facial area through a third-person view camera (120). The control unit (100) uses a deep learning-based facial recognition algorithm to identify anatomical landmarks that are stable and have little movement, such as eyes, nose, and jaw, within the image or 3D data.

[0134] Coordinate system generation: The control unit (100) generates a virtual coordinate system based on the three-dimensional positional relationship of multiple identified feature points. For example, a line connecting the center points of both eyes can be set as the X-axis, and the center of the glabella as the origin. The coordinate system thus generated serves as a fixed reference only for the patient, independently of the robot arm's base coordinate system.

[0135] Relative Position Registration: When a global coordinate system is established, the control unit (100) converts and registers the base position of the multi-axis robot arm (300) based on the global coordinate system. This allows the system to accurately recognize the relative position of the robot arm to the patient.

[0136] This allows the system to automatically, quickly, and consistently set up the working environment without the operator having to manually adjust the robot's position to the patient every time. Even if the patient is seated in a slightly different position within the dental unit, the system establishes a coordinate system based on the patient's face, ensuring that scanning can always begin with a consistent reference. Furthermore, it can also be utilized as a reference point to detect and compensate for subtle head movements during scanning.

[0137] One of the features of the global coordinate system according to the present invention is that it is set independently of the base coordinate system of the multi-axis robot arm (300) in order to ensure accuracy and robustness of scanning.

[0138] Specifically, the global coordinate system is set by being anchored to specific anatomical features of the patient (10) recognized by the third-person view camera (120). This means that the reference point of the system is the patient himself, who is the subject of the scan, rather than the robot or the surrounding environment.

[0139] If the global coordinate system is fixed to the base of the robot arm, serious errors will occur in the generated 3D model if the patient moves their head slightly during scanning. However, if the global coordinate system is established based on the patient's stable anatomical features (e.g., glabella, nose tip, cheekbones, etc.) as in the present invention, the reference point of the system will also move together even if the patient makes minute movements.

[0140] The control unit (100) identifies and corrects the relative transformation relationship between this global coordinate system fixed to the patient and the base coordinate system of the robot arm in real time. As a result, the system can perform robust scanning of the patient's movements and can always generate a consistent and accurate 3D model based on the patient's anatomical structure without the need for a separate complex head fixation device.

[0141] In the present invention, the control unit (100) performs forward kinematics calculations based on kinematic information of a multi-axis robot arm (300) to calculate the accurate position and orientation, i.e., pose, of the oral scanner (200) in three-dimensional space.

[0142] The core of the above kinematic information is the current joint angle of each of the multiple joints constituting the multi-axis robot arm (300). Each joint axis of the multi-axis robot arm (300) is equipped with a high-resolution encoder to measure the rotation angle of the corresponding joint in real time with very high precision. The control unit (100) continuously receives a data set containing the angle values ​​of all joints from these encoders. Since these values ​​are physical measurements, the problem of error accumulation, as in SLAM, does not occur.

[0143] Forward kinematics is a mathematical process of calculating the position and orientation of the robot arm end-effector relative to the robot's base coordinate system when the angle values ​​of each joint of the robot are known. The control unit (100) has a predefined kinematic model of the robot arm (length of each link, relative position between joints, etc.).

[0144] The control unit (100) performs forward kinematics calculations by inputting joint angle values ​​received from the encoder into this kinematic model. As a result of this calculation, precise pose information (410) representing the three-dimensional position (X, Y, Z) and three-dimensional attitude (rotation) of the robot arm end is calculated. Since this process is a deterministic calculation based on physical measurements rather than an estimation, it guarantees very high accuracy and reliability.

[0145] Finally, the control unit (100) determines the final pose of the oral scanner (200) sensor by applying a pre-corrected hand-eye transformation matrix to the pose of the robot arm end calculated in this way. The 3D model generator (400) can generate a 3D model in which accumulated error is fundamentally eliminated by aligning the 3D scan data using this pose information.

[0146] Forward kinematics calculations accurately calculate the pose of the end-effector coordinate system of the multi-axis robot arm (300). However, 3D data is acquired based on the sensor coordinate system of the oral scanner (200) coupled to the end of the robot arm. Since there is a fixed geometric displacement (translation and rotation) between these two coordinate systems, the final pose of the scanner sensor can only be determined by knowing this accurately.

[0147] To solve this problem, the present invention performs a pre-calibration process called hand-eye calibration when installing a system or replacing a scanner.

[0148] Hand-eye correction is a process of finding a fixed relative transformation relationship between the robot's 'hand (end of the robot arm)' and 'eye (scanner sensor).' Specifically, the control unit (100) controls the robot arm so that the oral scanner (200) captures a fixed correction pattern (e.g., a checkerboard) at various different positions and postures. The control unit (100) collects pose information of the robot arm end at each point of capture and position information of the correction pattern captured by the scanner. Using the collected multiple data pairs, the control unit (100) calculates a relative transformation matrix between the robot arm end coordinate system and the scanner sensor coordinate system, i.e., a hand-eye transformation matrix. This matrix is ​​calculated once and then stored in the system.

[0149] During the actual scanning process, the control unit (100) calculates the final pose of the oral scanner (200) in real time through the following operation.

[0150] The transformation matrix of the robot arm end-point coordinate system with respect to the robot base coordinate system is calculated through forward kinematics operations.

[0151] Load the pre-saved hand-eye transformation matrix (transformation matrix of the scanner sensor coordinate system for the robot arm end-system coordinate system).

[0152] By multiplying the transformation matrix of the robot arm end-point coordinate system with respect to the robot base coordinate system with the hand-eye transformation matrix, the transformation matrix of the scanner sensor coordinate system with respect to the robot base coordinate system, i.e., the precise final pose of the oral scanner, is finally calculated.

[0153] The final pose calculated through this process is directly used for 3D data alignment and plays a key role in ensuring the overall accuracy of the system.

[0155] The control unit (100) of the present invention can generate a final 3D model for the entire scan path of the oral scanner (200) without performing complex post-processing steps such as loop closure detection or pose graph optimization, which were essential in conventional SLAM-based systems.

[0156] In conventional SLAM-based scanning methods, cumulative error inevitably occurs as the scan path lengthens, causing distortion of the 3D model. To correct this distortion, the system had to re-recognize locations previously visited by the scanner (loop closure detection) and perform a mathematical optimization process (pose graph optimization) to globally minimize the error of the entire scan path based on this. This process required a significant amount of computation, necessitating a long waiting time to verify the final model even after the scan was completed.

[0157] The present invention does not use SLAM-based pose estimation, which is the root cause of accumulated error. Instead, it directly calculates the pose of the oral scanner (200) using kinematic information of a multi-axis robot arm (300) in which error does not accumulate. Each 3D scan data is accurately aligned to a globally consistent coordinate system from the beginning.

[0158] Therefore, since the accumulated error itself hardly occurs, the loop closure detection and pose graph optimization processes for correction become fundamentally unnecessary or their necessity is significantly reduced.

[0159] This allows users to view the final 3D model immediately after the scan is completed without any waiting time for post-processing, significantly improving the efficiency and speed of treatment. In addition, since it does not rely on complex loop closure algorithms that are prone to failure on tooth surfaces lacking feature points, the system operates more stably and reliably.

[0161] The three-dimensional model generated in the present invention is formed by accumulating three-dimensional data sequentially acquired from an oral scanner (200) on a global coordinate system. A three-dimensional point cloud or a voxel grid method can be typically used as a data structure to represent this three-dimensional model.

[0162] A point cloud is a data format that represents a three-dimensional shape as a set of points in three-dimensional space.

[0163] In the present invention, the oral scanner (200) generates a 'local point cloud' composed of numerous three-dimensional points every frame. The three-dimensional model generator (400) of the control unit (100) converts this local point cloud into a global coordinate system using precise pose information (410) calculated from the kinematic information of the robot arm. By continuously merging this converted point cloud into a 'global point cloud' in which previously scanned data is accumulated, a final three-dimensional model is completed. Each point may include color (RGB) information along with three-dimensional position (X, Y, Z) information.

[0164] A voxel grid is a data format that manages three-dimensional space by dividing it into small cubic units called 'voxels (Voxel, Volume + Pixel).'

[0165] When using this method, the control unit (100) sets the scan target area as a virtual three-dimensional grid. When new three-dimensional data is acquired from the oral scanner (200), the voxel state of the location to which the data belongs is updated. For example, each voxel may have a state value such as 'area not yet scanned', 'empty space', 'area where a surface exists', and may store additional information such as the average color of the surface or a normal vector.

[0166] The voxel grid method is efficient for integrating data and removing noise, and can be usefully applied to planning scan paths by identifying areas that have not yet been scanned.

[0168] The automatic oral scanning system of the present invention further includes an emergency stop means to ensure safety, as the multi-axis robotic arm (300) operates at a very close distance to the patient (10).

[0169] The above-mentioned emergency stop means is a device designed to immediately stop the operation of the system in the event of an unexpected malfunction or emergency. It can typically be implemented in the form of a large, red, mushroom-shaped push button that can be easily pressed by an operator.

[0170] It is desirable to place these emergency stop buttons in at least two locations. One is located on the console of the control unit (100) or on a separate control panel that the operator running the system can access immediately. The other is placed in an easily accessible location, such as a portable switch held in the patient's hand or on the armrest of the unit chair (20), so that the patient (10) can activate it directly. This allows the patient to stop the system immediately on their own if they feel discomfort or an abnormality during scanning, enabling them to proceed with the procedure with a sense of psychological stability.

[0171] When the emergency stop means is activated (when the button is pressed), a hardware interrupt signal that prioritizes ignoring the normal control logic of the control unit (100) is directly transmitted to the controller of the multi-axis robot arm (300). This signal causes the motor power of the robot arm joints to be cut off immediately or the brakes of all joints to be applied immediately.

[0172] As a result, the multi-axis robot arm (300) immediately stops all movements currently being performed at that location. The system, having entered an emergency stop state, does not resume any movements until the operator confirms safety and resets the system according to a set procedure.

[0173] These emergency stop measures perform an essential function in ensuring the safety of patients and operators in unpredictable situations, such as system malfunctions or unexpected patient behavior.

[0175] The control unit (100) of the present invention includes a deep learning-based policy model (110) to autonomously generate a scanning path and determine the operation of a robot arm without human intervention. This policy model mimics the scanning skills and judgment abilities of a skilled dentist or dental technician and serves as the core brain of the system that implements automation.

[0176] The policy model (110) can be described as a function designed to output an optimal action when a specific state is given.

[0177] Input (State): The policy model receives multi-modal data to comprehensively determine the current situation. This data includes real-time images of the inside of the oral cavity received from the oral scanner (200), images of the external environment received from the third-person view camera (120), and state information of the 3D model restored so far (e.g., areas where scanning is complete, areas that are missing, etc.) fed back from the 3D model generator (400).

[0178] Processing: The input high-dimensional data is processed by passing through a Deep Neural Network that constitutes the policy model. The neural network analyzes complex patterns and correlations within the data to form a deep understanding of the current scan state.

[0179] Output (Action): Based on the analyzed state, the policy model determines and outputs the next action to scan the entire dentition most efficiently and completely. This action is expressed in the form of specific values ​​for controlling the multi-axis robot arm (300), such as the target angle at which each joint of the robot must move or the relative amount of movement from the current position (delta pose).

[0180] This intelligent judgment ability is formed through prior learning using large amounts of data. The following methods can be used for learning.

[0181] Imitation Learning: Demonstrates the process of scanning multiple oral models by having an expert directly operate a robotic arm remotely or provide direct teaching. The system constructs a dataset of the states during this process (e.g., camera images) and the expert's actions (robotic arm operation values), and trains the policy model to mimic the expert's actions exactly under given conditions.

[0182] Reinforcement Learning: A policy model is allowed to perform scanning autonomously within a virtual simulation environment. In this process, rewards are assigned for desirable behaviors, such as scanning new areas, while penalties are given for undesirable behaviors, such as scanning areas that have already been scanned. The policy model learns by automatically updating its behavioral criteria (policy) in a direction that maximizes cumulative rewards.

[0184] The control unit (100) of the present invention preferably includes a GPU (Graphics Processing Unit) to accelerate computation so that the deep learning-based policy model (110) can operate smoothly in real time.

[0185] Deep learning models such as the policy model (110) operate based on parallel operations such as numerous matrix multiplications. In particular, in the case of the present invention, which must immediately determine the next movement of the robot arm by receiving real-time camera images and 3D data, a vast amount of calculation must be completed in a very short time.

[0186] A standard CPU (Central Processing Unit) is capable of handling complex sequential tasks but is inefficient at processing such large-scale parallel operations. If a policy model is run using only a CPU, it can take hundreds of milliseconds (ms) or more to determine a single action, which causes the robot arm's movements to become choppy or delayed, making smooth and safe scanning impossible.

[0187] A GPU is hardware specialized for large-scale parallel processing that incorporates thousands of small cores to process numerous simple operations simultaneously. By equipping the control unit (100) with a high-performance GPU and configuring the policy model (110) to perform operations on the GPU, the inference speed of the model can be improved by tens to hundreds of times compared to a CPU.

[0188] Through this, the control unit (100) can determine the next action from the policy model at a very fast speed within tens of milliseconds (ms). As a result, the multi-axis robot arm (300) can move smoothly and precisely in response to changes in the surrounding environment in real time, thereby greatly improving the speed, stability, and safety of the entire scanning process.

[0190] The policy model (110) of the present invention adopts a multi-modal input method that receives and processes multiple forms of data simultaneously, rather than a single type of data, similar to how a human expert synthesizes various information to make a judgment. This enables the robot arm to perceive the situation and generate actions more accurately and intelligently.

[0191] The main input data processed by the policy model (110) is as follows.

[0192] Image from the third-person view camera (120): This data provides 'Global context' information. That is, it provides a wide view of the entire scanning environment, such as the current position of the robot arm, the relative position of the scanner to the patient's face, and the patient's head movements. This plays an important role in ensuring macroscopic safety, such as oral entry or exit and collision prevention.

[0193] Image from the oral scanner (200): This data provides 'local context' information. That is, it provides real-time conditions of the fine inside of the oral cavity, such as the texture, shape, and boundaries of the tooth surface right in front of the scanner sensor. This plays a key role in microscopic control to precisely determine the next scan point and acquire high-quality 3D data.

[0194] 3D model data restored to date: This data serves as a 'temporal context' or 'memory'. The control unit (100) continuously feeds back to the policy model (110) information from the 3D model generator (400) about which tooth regions have been scanned with sufficient quality to date, which regions are missing (holes), or which have low data density.

[0195] The policy model (110) fuses and processes these heterogeneous data within a deep neural network. For example, the policy model (1) recognizes that the scanner is near a molar through an external camera, (2) identifies specific occlusal surface features of the molar through an internal scanner image, and (3) confirms that there is still insufficient data of the occlusal surface through 3D model information.

[0196] Through such comprehensive judgment, the policy model can go beyond simply moving to the next point and generate much more sophisticated and situationally appropriate optimal actions, such as 'tilting the scanner angle slightly more to approach and scan the occlusal surface of the molar where data is missing.' This ability to process multimodal input is a key element that enables the automatic scanning system of the present invention to have a high level of autonomy and intelligence.

[0198] The policy model (110) of the present invention is designed to receive data of a specific form, particularly during the intraoral scanning step, and to output data of a specific form that can directly control the robot arm.

[0199] While the scanning loop (700) is in progress, the policy model (110) continuously receives the following two key pieces of information to accurately determine the current state.

[0200] Real-time image acquired from the oral scanner (200): This provides the policy model with high-resolution local information regarding the fine shape, texture, and boundary with adjacent teeth of the tooth surface currently being viewed by the scanner.

[0201] 3D models restored to date: This provides the policy model with global information about the overall progress of the scan. Through this information, the policy model can clearly distinguish between areas where sufficient data has already been secured and areas where data is still insufficient or missing.

[0202] The policy model (110) calculates and outputs the optimal action that the multi-axis robot arm (300) should take for the next scan based on the input data. In a preferred embodiment of the present invention, this action is defined as a set of target joint angle values ​​for each of the plurality of joints of the multi-axis robot arm (300).

[0203] For example, in the case of a 6-axis robot arm, the policy model outputs a vector consisting of 6 real values ​​such as (θ₁, θ₂, θ₃, θ₄, θ5, θ6). The control unit (100) receives these target joint angle values ​​and directly controls each joint of the robot arm to reach the corresponding angle.

[0204] This approach means that the policy model directly generates the physical posture of the robot arm required for scanning. This minimizes complex intermediate transformation processes (e.g., inverse kinematics calculations), thereby increasing the response speed of the control loop and enabling the effective learning of the intuitive relationship (end-to-end mapping) between visual information and the robot's physical movement during the learning process.

[0206] In another preferred embodiment in which the policy model (110) outputs the motion of the multi-axis robot arm (300), the motion may be defined as a delta pose including a movement displacement and rotation relative to the current pose of the oral scanner (200).

[0207] Delta pose is a method that defines "how much more to move and rotate from the current state" instead of directly outputting the absolute target joint angles of the robot arm. This can be expressed as a 6-dimensional vector containing 3-dimensional relative displacement (Δx, Δy, Δz) and 3-dimensional relative rotation (ΔRx, ΔRy, ΔRz).

[0208] When using this method, the control process of the control unit (100) is as follows.

[0209] Delta Pose Generation: The policy model (110) calculates and outputs the next scanner's relative movement, i.e., the delta pose, which is most advantageous for scanning, based on the current input state.

[0210] Target pose calculation: The control unit (100) calculates a new absolute pose of the next target point by applying (synthesizing) the delta pose output by the policy model to the current absolute pose of the oral scanner calculated through forward kinematics.

[0211] Inverse Kinematics Calculation: The control unit (100) calculates the target angle values ​​of each joint required to move the robot arm end to the calculated new absolute pose through inverse kinematics calculation.

[0212] Robot arm control: The control unit (100) drives the robot arm according to the calculated target joint angle value.

[0213] Motion generation methods using delta poses can have the following advantages.

[0214] Smooth Scan Path Generation: Rather than specifying a new absolute target point every time, a method of commanding a gradual change from the current state is advantageous for generating a more continuous and smooth scan path. This has a positive effect on acquiring high-quality 3D data.

[0215] Learning Efficiency: It may be easier for a policy model to learn to predict the next small change from the current state rather than predicting the entire pose of a complex robot all at once, which can increase the model's learning efficiency and stability.

[0217] The policy model (110) of the present invention may be composed of two independent sub-policy models with clearly distinct roles to perform the entire automatic scanning process more efficiently and stably. This is an approach to solving the problem by dividing the overall problem into smaller and easier-to-manage problems.

[0218] Policy 1: Oral Entry

[0219] The first policy is a model specialized in generating oral entry actions, which is the first step of scanning.

[0220] Main task: to safely and accurately move the oral scanner (200) from the initial position of the robot arm to the starting point of the scan inside the patient's oral cavity.

[0221] Main Input: This policy primarily uses video from a third-person view camera (120) as input. Through video from an external view, the position of the patient's face and mouth and obstacles are identified macroscopically to plan a collision-free entry path.

[0222] Activation time: The first policy is activated at the entry stage (600) of Fig. 3, and its role ends when the scanner successfully enters the oral cavity.

[0223] Policy 2: Internal Scanning

[0224] The second policy is a model specialized in generating an intraoral scanning motion that scans the actual tooth surface after entering the oral cavity.

[0225] Primary mission: To generate the optimal scan path in real time to scan the entire dentition completely and efficiently.

[0226] Main Input: This policy primarily uses real-time data from the oral scanner (200) and the 3D model generated so far as inputs. By directly observing the fine shape inside the oral cavity, it determines the areas that have already been scanned and the areas that have not yet been scanned to determine the next action.

[0227] Activation time: The second policy is activated in the scanning loop (700) of FIG. 3 after the first policy is terminated and repeatedly generates actions until the entire scan is completed.

[0228] By separating the policy model into two specialized roles in this way, each model can learn and operate by focusing solely on its own task. This simplifies the learning process compared to cases where a single model must handle both complex tasks, and maximizes the performance and stability of each stage.

[0230] The first policy of the present invention actively utilizes three-dimensional depth information obtained from a third-person view camera (120) to maximize the safety of the system during the oral entry stage. Through this, the first policy creates a safe path that actively avoids physical collision between the patient's face and the oral scanner (200), rather than a path that simply enters the oral cavity.

[0231] When the first policy is activated, the control unit (100) receives real-time 3D point cloud data from a third-person view camera (120, preferably an RGBD camera). Based on this data, the control unit (100) constructs a 3D virtual environment model in real time that includes objects existing within the workspace, namely the patient's facial parts (nose, chin, cheeks, lips, etc.), the oral scanner (200), and parts of the multi-axis robotic arm (300).

[0232] The first policy sets a starting point (current position of the robot arm) and a target point (scan starting point inside the oral cavity) within this 3D virtual environment model and searches for an optimal movement path connecting the two points. At this time, the first policy continuously verifies whether the 3D model of the oral scanner (200) geometrically interferes (collides) with the 3D model of the patient's face at all points along the path.

[0233] If a collision is predicted on the explored path, the first policy re-searches for an alternative path where no collisions occur. This process is repeated until a safe path with absolutely no risk of collision is finally secured.

[0234] Thanks to this collision avoidance path generation capability, the system of the present invention can safely perform oral entry operations while always preventing physical contact with the patient, regardless of the patient's facial shape or posture. This is a key function that ensures the safety and reliability required of an automated medical robot system.

[0236] The second policy of the present invention may be configured to generate an action by receiving not only data from the oral scanner (200) but also images acquired from the third-person viewpoint camera (120) together, in order to further increase the stability and accuracy of the oral cavity scanning.

[0237] The oral scanner (200) provides precise local information of the 'close view' of the tooth surface being scanned. On the other hand, the third-person view camera (120) provides global information of the 'far view' including the overall position of the scanner, the patient's face, and surrounding soft tissues (tongue, cheeks).

[0238] The second policy fuses information from these two perspectives, acquiring a much higher level of situational awareness, much like a person looking at their own hand while simultaneously being conscious of their surroundings.

[0239] Specific applications and effects

[0240] The fusion of such multi-point information provides the following specific advantages.

[0241] Prevention of collision with soft tissue: Because the field of view of an oral scanner is narrow, it is difficult to detect when the scanner body is approaching the tongue or cheek. The second policy detects dangerous proximity between the scanner and soft tissue through video from a third-person camera and finely corrects the path before a collision occurs to minimize patient discomfort and ensure safety.

[0242] Robust response to patient movement: If the patient moves their head during scanning, the third-person camera can immediately detect this movement. Based on this information, the second policy pauses the scanning or replans the robotic arm's path in real time to compensate for the movement. This prevents the scan from failing or data quality from degrading due to patient movement.

[0243] In conclusion, by using two types of camera inputs together, the second policy enables the generation of safer, more robust, and intelligent internal scanning operations in real clinical environments where unpredictable situations may occur.

[0245] The second policy of the present invention performs real-time safety monitoring and motion adjustment functions using images from a third-person view camera (120) to intelligently respond to the dynamic and unpredictable environment of intraoral scanning.

[0246] In the oral cavity, there are soft tissues such as the tongue, cheeks, and gums, in addition to the teeth. The field of view of the oral scanner (200) is focused on the surface of the tooth being scanned, so it is difficult to perceive that the body of the scanner is unnecessarily touching or pressing on these soft tissues.

[0247] To address this, the second policy continuously analyzes the image from the third-person view camera (120) to calculate the distance between the 3D model of the oral scanner (200) and the patient's soft tissue in real time. If this distance becomes too close to a preset safety threshold and a collision risk is detected, the second policy immediately fine-tunes the movement of the robotic arm. For example, by slightly changing the angle of the scanner or slightly diverting the path to avoid contact with the soft tissue, the patient's discomfort is minimized and the gag reflex is prevented.

[0248] Unexpected movements, such as the patient coughing or moving their head, may occur during scanning. These movements can compromise the accuracy of the scan data or cause safety issues.

[0249] The second policy tracks changes in the patient's head position in real time relative to the global coordinate system set during the initialization phase through a third-person view camera (120). If movement outside the set range is detected, the second policy intelligently adjusts the movement of the robot arm as follows, depending on the severity of the situation.

[0250] Fine movement: Corrects the scan path in real-time to match the patient's new position, allowing for continuous scanning without interruption.

[0251] Large or sudden movement: Immediately pause scanning and quickly retract the multi-axis robotic arm (300) to a safe standby position. Afterward, when the patient stabilizes again, resume scanning or restart from that point with the operator's confirmation.

[0252] Through these functions, the system of the present invention realizes safe and robust automated scanning that goes beyond simple repetitive tasks and actively responds to the variability of actual clinical environments.

[0254] As an alternative embodiment of the separated policy model described above, the policy model (110) of the present invention can be implemented as a single, integrated model that performs both oral entry and internal scanning.

[0255] This single integrated model is a deep neural network responsible for the entire scanning process from start to finish. The model is trained to vary the types of input data primarily used depending on the stage of the task.

[0256] Entry phase: When the scanning process begins, the integrated model focuses primarily on images from the third-person view camera (120). Based on this external view information, it identifies the patient's face and oral position and sequentially generates collision-free oral entry movements. At this stage, the data from the oral scanner (200) is rarely used or is utilized only as an auxiliary tool.

[0257] Scanning step: When the oral scanner (200) successfully enters the oral cavity, the model naturally switches the primary input source to the data from the oral scanner (200) and the currently restored 3D model. Based on this internal viewpoint information, an internal scanning motion is sequentially generated to efficiently cover the entire dentition.

[0258] As such, the single integrated model operates by independently determining, through its internal context, whether the current task is in the entry or scanning phase, and selecting and utilizing the appropriate input data.

[0259] This single integrated model approach can have the following advantages.

[0260] Natural motion transition: Since there is no process of transferring control between the two models, a very smooth and seamless transition from oral entry to internal scanning is possible.

[0261] Optimization of the entire process: A single model can learn the interrelationships between the entry and scanning processes. For example, it can perform optimization across the entire process by autonomously learning that entering in a specific manner is more advantageous for scanning specific teeth.

[0262] Structural simplicity: Since only one model needs to be managed without the need to develop and manage two models separately, the overall structure of the system can be simplified in some cases.

[0264] The policy model (110) of the present invention goes beyond simply generating a scan path and performs an intelligent role of determining when to complete the scanning and safely terminating the process.

[0265] Inside the control unit (100), pre-set quality standards that a successful three-dimensional model must possess are defined. These standards may include the following items.

[0266] Coverage: Whether all surfaces of the entire dentition (maxilla or mandible) have been scanned without any missing parts. In other words, there must be no holes larger than a specified size within the 3D model.

[0267] Data Density: Whether each tooth surface is represented by sufficiently dense 3D points. This is to ensure the precision required for fabricating prosthetics, etc.

[0268] Data Quality: Whether blur caused by noise or motion in the scanned data has been minimized.

[0269] While the scanning loop (700) is in progress, the policy model (110) continuously receives the currently generated 3D model from the 3D model generator (400) and evaluates in real time whether it satisfies the quality criteria.

[0270] When the policy model (110) determines that the current model meets all quality criteria, it makes a decision of "scan complete" and declares the end of the scanning loop (700).

[0271] When the termination of the process is determined, the policy model (110) generates an exit action that finally moves the oral scanner (200) to a safe initial position outside the patient's (10) oral cavity.

[0272] At this time, the policy model (110) calculates a safe exit path that does not collide with the patient's teeth, lips, facial area, etc. by utilizing information from the third-person view camera (120), just as when entering the oral cavity. The control unit (100) controls the multi-axis robot arm (300) along this path to smoothly withdraw the scanner, and then finally terminates the entire scanning process (820).

[0273] These automatic shutdown and exit functions play an important role in objectively guaranteeing the completeness of the scan and consistently and safely completing the entire process.

[0275] The policy model (110) of the present invention performs a hole filling function to actively improve the completeness of the scan, going beyond simply passively checking pre-set quality standards.

[0276] When the scanning process reaches its final stage, the control unit (100) performs a detailed analysis of the three-dimensional model created so far under the control of the policy model (110). This analysis examines the surface of the model to identify areas where data is completely missing or areas of poor quality where the density of points is significantly lower than a set threshold. These incomplete data areas are identified as 'hole areas' and their locations and ranges are specified.

[0277] Hole areas may occur due to undercuts, areas where data acquisition was difficult due to the scanner's approach angle, diffuse reflection of light, or saliva.

[0278] The policy model (110) first checks whether there is an identified hole area before deciding to end the scan process.

[0279] If a hole area exists, the policy model (110) defers the termination decision and generates an additional scan path aimed at filling that hole area. This path is not an inefficient method of rescanning the entire dentition. Instead, it is a targeted short path that calculates the optimal scanner position and orientation to best acquire data for a specific hole area and precisely moves the robot arm to that point.

[0280] As the robot arm moves along this additional path and acquires data, the new data is integrated into the existing 3D model to fill in the hole area.

[0281] After the hole filling operation is completed, the policy model (110) re-evaluates the overall quality of the 3D model once again. If another hole area is found, an additional scan path is regenerated to fill that area.

[0282] This cyclical process of 'inspection > discovery > correction' is repeated until no more major hole areas are found in the model and all quality standards are satisfied. Through this process, the system of the present invention can ensure a final 3D model with a very high level of completeness, just as a skilled operator meticulously re-checks deficiencies and finishes the scan.

[0284] The policy model (110) of the present invention is a machine learning model that does not operate according to fixed rules but learns optimal behavior criteria from data on its own. For this learning, reinforcement learning or imitation learning methodologies may be applied.

[0285] Reinforcement learning is a method in which a policy model learns the optimal behavioral strategy on its own through trial and error.

[0286] Learning Environment: Learning primarily takes place in a virtual 3D simulation environment implemented to closely resemble reality. This environment contains various types of dental models, as well as virtual robotic arms and scanners.

[0287] Learning Process: The policy model (agent) autonomously performs scanning actions within the simulation environment. As a result of performing a specific action, it receives a positive reward if it obtains a positive outcome, such as "scanning new areas that were not previously scanned," and a negative reward if it obtains a negative outcome, such as "duplicate scanning of areas that have already been scanned."

[0288] Goal: The learning objective of the policy model is to maximize the total sum of accumulated rewards upon completing a single scanning process. Through numerous iterative simulations, the policy model learns on its own which action to take in which situation to obtain the highest accumulated reward—that is, the optimal scanning policy.

[0289] Imitation learning is a method in which a policy model learns behavioral strategies by mimicking demonstration data from human experts.

[0290] Data Collection: First, a professional, such as a skilled dentist or dental technician, demonstrates the process of scanning dental models of various cases multiple times using a teleoperation device or by direct teaching the robot arm. The control unit (100) builds a large dataset by pairing the situation (State, e.g., camera image, 3D model state) that the professional encountered during this process with the action (Action, e.g., robot arm joint angle control value) that the professional performed in that situation.

[0291] Learning Process: The policy model is trained using the constructed expert dataset. The learning objective is to optimize the model's internal parameters to output an action most similar to the action performed by an expert when given State data as input. In other words, the policy model is trained to replicate the expert's know-how and scanning techniques exactly. For example, modern techniques such as Diffusion Policy can be applied for this imitation learning.

[0293] The method of training the policy model (110) of the present invention is preferably performed using an offline learning method to ensure the safety and stability of the system.

[0294] Unlike online learning, where a policy model learns by interacting with the environment in real time, offline learning is a method that completes training using only a pre-collected static dataset. The training process is clearly separated into a data collection phase and a model training phase as follows.

[0295] Dataset Construction Phase: Before starting training, a large-scale dataset is constructed in advance. This dataset can be an expert demonstration dataset (for imitation learning) or a scanning dataset (for offline reinforcement learning) containing logs collected from scanning performed with various policies in a simulation environment.

[0296] Model Training Phase: The policy model is trained using only the constructed dataset. During this process, the model does not move the robot arm in real time. Once training is complete and the model's performance is sufficiently verified, the trained parameters are frozen.

[0297] Deployment Phase: The verified policy model is loaded into the control unit (100) of the actual automatic oral scanning system and used. During the actual scanning operation, learning is no longer performed, and only inference based on the learned results is performed.

[0298] Due to the nature of the present invention, which operates a robot in a medical environment, the offline learning method provides the following important advantages.

[0299] Ensuring Safety: Fundamentally eliminates the possibility of the robot performing unexpected and dangerous actions during the learning process. All learning is conducted solely using data in a safe environment that does not affect actual patients or equipment.

[0300] Stability and Predictability: Once deployed, the behavior of the policy model does not change. This is crucial for ensuring the stability and predictable operation required of a medical device.

[0301] Reproducibility and Ease of Verification: Training with the same dataset consistently yields a model with identical performance, guaranteeing the reproducibility of training results. This is advantageous for objectively verifying and certifying the system's performance.

[0303] To increase the generalization performance and robustness of the policy model (110), a data augmentation technique is applied to the learning process to artificially increase the collected limited learning dataset.

[0304] The performance of deep learning models depends heavily on the quantity and diversity of training data. However, it is practically impossible to collect expert demonstration data or scanning data for every possible clinical situation (various oral structures, lighting conditions, camera angles, etc.). Models trained with only limited data may experience overfitting, a problem where performance deteriorates rapidly when faced with real-world situations that differ only slightly from the training data.

[0305] Data augmentation is a technology that applies various transformations to existing collected data to produce the effect of collecting new data. Through this, it trains policy models to flexibly respond to various changes in the real environment.

[0306] The following augmentation techniques can be applied to the image data (third-person view camera and oral scanner images) that serve as input to the policy model.

[0307] Geometric Augmentation:

[0308] Rotation, Scaling, Translation: Images are rotated slightly, enlarged or reduced, or moved left / right and up / down. Through this, the model learns the ability to recognize an object identically even when viewed from slightly different angles or distances.

[0309] Photometric Augmentation:

[0310] Brightness / Contrast Jittering: Randomly changes the brightness or contrast of the image. This helps create a model robust to changes in lighting in the examination room.

[0311] Color Jittering: Finely alters the saturation, hue, etc., of an image to support various color tones.

[0312] Noise Injection: Random noise, such as Gaussian noise, is added to the image to improve the model's stability against unpredictable distortions, such as camera sensor noise.

[0313] These transformations are applied only to the input data, the 'State,' while the expert's 'Action' data corresponding to that state remains unchanged. In other words, the model learns that "even if the appearance changes slightly, the action to be taken is the same."

[0314] Through this data augmentation technique, the policy model (110) can achieve excellent generalization performance even with limited data, and can perform stable and reliable operations even in actual clinical environments encountered for the first time.

[0316] The imitation learning of the present invention utilizes a Diffusion Policy model as a cutting-edge technique for learning expert scanning skills more precisely and naturally. The Diffusion Policy applies the principles of the Diffusion Model, which is widely used in the field of Generative AI, to robot control.

[0317] Learning Probability Distributions (Forward Process): First, during the training phase, the diffusion policy utilizes an expert demonstration dataset. Specifically, it learns a process (Forward Process) that gradually adds random noise to data regarding the optimal actions performed by experts in specific situations, eventually transforming them into complete white noise. Through this process, the model learns which probability distribution the experts' sophisticated actions follow.

[0318] Action Generation from Noise (Reverse Process): When generating actions in actual scanning situations, the reverse process of the learning process is followed. The policy model first starts with completely random noise. Then, referencing the learned probability distribution, it repeatedly performs a denoising process to progressively remove noise from this noise. As the final result of this process, clean and sophisticated action data best suited to the current situation is generated.

[0319] Applying this diffusion policy to imitation learning yields the following significant advantages.

[0320] Natural and smooth behavior generation: The diffusion policy does not replicate expert behavior as a single value, but rather learns the overall data distribution. The behavior generated through this process traces a very natural and smooth trajectory, just like a human moving, thereby increasing the movement stability of the robot arm.

[0321] Diverse Situation Handling Ability: Since behaviors are generated through sampling within a learned probability distribution, it is possible to create behaviors that respond flexibly and in various ways to similar situations, even if they are not exactly identical. This enhances the model's robustness against subtle changes in real-world clinical environments.

[0322] High Expressiveness: It can model complex and subtle expert scanning know-how with high expressiveness, enabling high-quality scanning that approaches the level of human experts, going beyond simple imitation.

[0324] In order to train the policy model (110) of the present invention using an imitation learning method, a process of first constructing a high-quality dataset containing human expert scanning technology is performed.

[0325] To collect data, a skilled expert demonstrates the scanning process using a remote operation system. This system consists of a master arm operated directly by the expert and a slave arm that moves in exactly the same way as the expert's movements. Here, the slave arm corresponds to the multi-axis robot arm (300) of the present invention.

[0326] The expert holds the master arm in a safe and convenient position and moves it naturally, as if scanning directly with their own hand. This movement is transmitted in real time to the slave arm, a multi-axis robotic arm (300), so that the oral scanner (200) at the end of the robotic arm precisely scans the oral model for demonstration.

[0327] The control unit (100) collects and synchronizes all relevant data indicating the scanning situation in real time while the expert is demonstrating. The collected data is as follows.

[0328] Third-person view camera footage: Video data capturing the entire robotic arm and oral model from the outside.

[0329] Oral scanner image: Close-up image or three-dimensional data of the inside of the oral cavity directly captured by the oral scanner (200).

[0330] Joint angle data of the robot arm manipulated by the expert: Angle values ​​of each joint that the slave arm, the multi-axis robot arm (300), actually moved as a result of the expert moving the master arm. This data corresponds to the 'correct action' selected by the expert in that situation.

[0331] The control unit (100) records accurate timestamps on these three types of heterogeneous data streams so that the 'situation (camera images)' at a specific point in time and the corresponding 'correct action (joint angle)' are accurately matched.

[0332] The data pairs collected and synchronized through these numerous demonstrations are constructed into a final dataset for training a policy model (110). Using this dataset, the policy model is trained to output the same joint angle output as an expert when a specific image input is given.

[0334] As an alternative to remote operation methods, expert data for imitation learning may be collected through direct teaching. Direct teaching is an intuitive method in which an expert demonstrates a desired movement by directly holding and moving the multi-axis robot arm (300) with their hands, without complex remote operation equipment.

[0335] Switching to Free Drive Mode: Before starting data collection, the control unit (100) switches the multi-axis robot arm (300) to a 'free drive mode' or 'gravity compensation mode'. In this mode, the robot arm maintains only a minimal force to offset its own weight, allowing a professional to smoothly move the robot arm to a desired position and posture with almost no effort.

[0336] Expert Demonstration: The expert directly grasps the end of the robot arm equipped with the oral scanner (200) and moves the robot arm along the optimal scanning path inside the oral model, just as if scanning a real patient.

[0337] Data synchronization and collection: While the robot arm is moving, the control unit (100) records, as in the remote operation demonstration, (1) third-person view camera footage, (2) oral scanner footage, and (3) real-time angle data of each joint of the robot arm (expert action) with timestamps in synchronization.

[0338] This method allows an expert to easily transfer their scanning know-how to the robot without the need for expensive master arm equipment. Since the expert directly moves the robot arm and teaches the path through physical senses, it is advantageous for collecting very intuitive and sophisticated demonstration data. The collected dataset is used to train a policy model (110) in the same way as the remote operation method.

[0340] The success of imitation learning depends on a dataset that accurately captures "what action was taken in a specific state." In this invention, the "state" is represented by multiple camera images and the "action" by the joint angles of a robot arm; these are asynchronous data streams output from different hardware devices at different intervals.

[0341] Accordingly, the control unit (100) performs a process of precisely synchronizing the data coming from various sources based on timestamps in order to make them meaningful learning data.

[0342] While the expert is demonstrating the scanning, the control unit (100) receives the following three main data streams in parallel.

[0343] Third-person view camera data stream: A series of video frames capturing the external environment.

[0344] Oral scanner data stream: Continuous video frames or 3D data capturing the inside of the oral cavity.

[0345] Multi-axis robot arm joint encoder data stream: Real-time angle values ​​of each joint based on expert movement.

[0346] The control unit (100) records a precise timestamp in microseconds on the data as soon as new data (e.g., one video frame, one set of joint angles) arrives from each stream.

[0347] When data collection is complete, the control unit (100) sorts and merges each data stream based on the recorded timestamps to create a final expert dataset.

[0348] Specifically, the control unit (100) finds the third-person view camera frame and the oral scanner frame with the closest timestamp to the robot arm joint angle data at a specific point in time and groups them into a single data pair.

[0349] If this process is repeated for all data, each item in the final dataset will have a structure such as {timestamp, third_person_view_image, oral_scanner_image, expert_joint_angle}.

[0350] Through this precise synchronization process, high-quality training data with a clear causal relationship regarding "what action (Action) an expert took when viewing specific images at a specific time t (State)" can be constructed, which serves as a basis for maximizing the learning performance of the policy model (110).

[0352] In the present invention, when training a policy model (110) using reinforcement learning, a key element for guiding the model's behavior in a desirable direction is a reward function. The reward function is a rule that evaluates how good a specific behavior of the policy model (agent) was, and in the present invention, it is sophisticatedly designed to maximize the coverage and data quality of the scanned tooth area.

[0353] The primary goal of scanning is to record every tooth surface without omission. To this end, the reward function grants a positive reward (+) for the behavior of exploring new areas.

[0354] Implementation Example: The control unit (100) calculates how much the total surface area or number of voxels of the 3D model has increased after the robot arm performs a specific action, based on the 3D model restored so far. If a new area that was not previously scanned has been added, a reward proportional to the increase is provided to the policy model.

[0355] Penalty: Conversely, a reward of 0 or a negative penalty is given for redundant actions of rescanning an area where sufficient data has already been secured.

[0356] Through this reward system, the policy model naturally learns to prioritize areas that have not yet been scanned, thereby mastering a strategy to efficiently cover the entire dentition.

[0357] Scanning goes beyond simply scanning all areas; it is crucial to obtain high-quality, clinically usable data. To this end, the reward function grants a positive reward (+) for actions that improve data quality.

[0358] Implementation example: The control unit (100) evaluates quality indicators of newly acquired 3D scan data in real time. Quality indicators may include data density, distance between the scanner and the tooth surface, and the degree of motion blur. If the behavior of the policy model contributes to obtaining clear and high-density data by maintaining optimal distance and speed, a reward proportional to the quality score is provided.

[0359] Penalty: A penalty is applied if the scanner is moved too quickly, resulting in blurry data, or if the data is inaccurate due to moving out of the optimal focal length.

[0360] In conclusion, the policy model learns with the goal of maximizing the sum of 'coverage rewards' and 'quality rewards'. Through this, it autonomously acquires the optimal scanning technology to obtain high-quality data at a fast pace, rather than being merely fast or thorough.

[0362] In order to effectively guide the behavior of a policy model (110) in reinforcement learning, a reward function that converts an abstract goal of ‘coverage maximization’ into a concrete numerical signal is designed as follows. This reward function consists of a combination of a reward for ‘new data acquisition’ and a penalty for ‘duplicate scans’.

[0363] Positive reward(+): Acquire new data

[0364] Immediately after the policy model performs a unit action (e.g., a step of a robot arm movement), the control unit (100) aligns the newly acquired 3D scan data through the oral scanner (200) with the existing global 3D model.

[0365] At this time, the control unit (100) quantitatively measures the amount of valid data among the newly added data. For example, it can calculate the number of 3D points newly added to the global 3D model or the number of voxels that have changed from an 'unscanned' state to a 'scanned' state.

[0366] The policy model receives a positive reward proportional to the amount of newly added valid data. In other words, the larger the previously unscanned area scanned in a single move, the greater the reward. This mechanism strongly motivates the policy model to actively explore unknown territories.

[0367] Negative compensation (-): Prevents duplicate scans

[0368] If the unit operation of the policy model moves the robot arm to rescan an area where data has already been secured with sufficient quality, the newly acquired data contributes little to the existing model. The control unit (100) determines that the case where there is little or no amount of newly added valid data is a 'duplicate scan'.

[0369] When such duplicate scans occur, the policy model receives a pre-set fixed value of negative reward, or penalty. This penalty serves to deter the policy model from inefficiently scanning the same area repeatedly.

[0370] In conclusion, the policy model learns in a direction that maximizes positive rewards and minimizes negative rewards. Through this, it autonomously constructs a highly efficient scanning strategy that systematically explores the entire dentition and minimizes overlap.

[0372] The reward used in the reinforcement learning of the present invention is calculated in direct linkage with accurate pose calculation using kinematic information, which is the core technology of the present invention. This is an important mechanism that guarantees the efficiency and accuracy of reinforcement learning.

[0373] The reward for performing a unit operation of the policy model (110) is calculated through the following sequential process.

[0374] Model state record: The control unit (100) records the state of the 3D model completed up to now (model before update, M_t) just before the robot arm performs the operation.

[0375] Precision pose-based model update: When the robot arm performs an action, the control unit (100) aligns the 3D scan data acquired during the corresponding action interval with the pre-update model (M_t) using the precise pose calculated from the kinematic information of the multi-axis robot arm (300). As a result, a post-update model (M_{t+1}) with new data added is generated.

[0376] Quantitative measurement of change amount: The control unit (100) quantitatively measures the difference between the model before the update (M_t) and the model after the update (M_{t+1}). This 'difference' corresponds to the 'amount of newly added valid data' described above, and specifically can be measured as the volume of newly filled voxels, the number of points in the added point cloud, or the increased surface area.

[0377] Compensation Value Calculation: Finally, the measured change in the model is input into the compensation function to calculate the final compensation value.

[0378] This method is effective in providing a very accurate and reliable reward signal to reinforcement learning.

[0379] If the model is updated with an inaccurate SLAM-based pose, the amount of change in the model will also be measured inaccurately, which gives an incorrect learning signal to the policy model. However, since the present invention uses an error-free kinematic pose, the amount of change in the model can be measured accurately.

[0380] Consequently, the policy model can receive clear feedback on how much its actions contributed to scan completion, enabling it to learn the optimal scanning strategy more quickly and effectively. This is a key aspect where the 'precise kinematics' and 'intelligent reinforcement learning' of this invention are organically combined to create synergy.

[0382] The three-dimensional model generated by the present invention is intended to be used directly for the fabrication of actual dental prostheses (e.g., crowns, bridges) or surgical guides, beyond mere visual appreciation. Therefore, the model must satisfy the high level of accuracy required for such precise subsequent work.

[0383] To this end, the three-dimensional model of the present invention is generated to satisfy a pre-set spatial resolution.

[0384] Spatial resolution is a measure of how detailed and precise a 3D model represents the actual shape.

[0385] In the case of a point cloud: spatial resolution is defined by point density, that is, how many three-dimensional points exist per unit area.

[0386] For a voxel grid: Spatial resolution is defined by the voxel size, and a smaller voxel size implies higher resolution.

[0387] This spatial resolution value is set in advance in the control unit (100) according to clinical purposes. For example, to produce a prosthetic margin line requiring precision in units of tens of micrometers (μm), the target resolution can be set to 50 μm.

[0388] The set target resolution serves as an important criterion for determining the scanning strategy of the policy model (110).

[0389] Path and speed planning: The policy model (110) calculates the scan speed, distance from the tooth surface, and overlap rate of the scan path that are most suitable for achieving the target resolution to generate the motion of the robot arm.

[0390] Quality evaluation criteria: During scanning, the control unit (100) continuously evaluates whether each region of the three-dimensional model satisfies the set resolution criteria. If the data density of a specific region falls short of the target value, that region is considered a 'hole region' and additional scanning is performed.

[0391] By clearly setting the target resolution and directly reflecting it in the scanning control, the present invention ensures that the finally generated 3D model has value as a metrically accurate digital model, going beyond simple shape data. This is an essential element that guarantees the success of subsequent CAD / CAM operations and the clinical success of the final prosthesis.

[0393] The three-dimensional model generated by the present invention can be generated as a model with a texture mapped with color (RGB) information combined to realistically represent the appearance of the inside of the actual oral cavity, going beyond simply including geometric shape information.

[0394] The oral scanner (200) used in the present invention simultaneously acquires depth information for a three-dimensional shape and color (RGB) information for the color of the surface.

[0395] The 3D model generator (400) of the control unit (100) fuses these two types of information when generating a 3D model. Specifically, for each 3D point (X, Y, Z) of the 3D point cloud generated from depth information, it finds and assigns the RGB value of the pixel corresponding to the same location in the color image. In the case of a voxel grid, it can store the average color value of the points included in the corresponding voxel.

[0396] Through this texture mapping process, the final 3D model generated becomes a photorealistic digital model with realistic colors and textures.

[0397] Including color information in three-dimensional models provides significant clinical value beyond simple visual effects, such as the following.

[0398] Accurate Shade Matching: When fabricating prosthetics such as crowns and veneers, it is very important to select a color that harmonizes with the surrounding natural teeth. Texture-mapped 3D models provide an objective and permanent digital record of the patient's tooth color, enabling much more accurate and consistent shade matching than manual color comparison.

[0399] Ease of tissue differentiation: Allows for clear differentiation of tooth and gum (gingival) tissues on the model. In particular, it enables clear visual identification of the gingival margin, which serves as the boundary of the prosthesis, thereby contributing significantly to more precise prosthesis design and fabrication.

[0400] Clinical Diagnosis Assistance: It assists in diagnosis and treatment planning by helping to visually identify various clinical conditions within the oral cavity, such as tooth discoloration, gum inflammation (redness) in specific areas, and existing restorations made of different materials.

[0402] The 3D point cloud or voxel grid generated by the present invention is close to raw data representing a 3D shape. The data format used as a standard in most dental CAD (Computer-Aided Design) software or 3D printers is a polygon mesh with a continuous surface.

[0403] Accordingly, the control unit (100) of the present invention further performs the step of finally converting the original data into a polygon mesh form to maximize clinical utility.

[0404] The control unit (100) applies a surface reconstruction algorithm to the final three-dimensional point cloud or voxel grid accumulated after scanning is completed.

[0405] Surface reconstruction is the process of generating a topologically continuous surface by estimating it from a set of discrete points. This can be likened to creating a smooth 'shell' by connecting points scattered in three-dimensional space. To this end, various algorithms can be used, such as Poisson Surface Reconstruction, the Ball-Pivoting Algorithm, or Marching Cubes, which are primarily used for voxel grids.

[0406] As a result of the above algorithm, the 3D model is finally generated in the form of a polygon mesh represented as a set of numerous small polygons (mainly triangles). This mesh model can be saved in standard 3D file formats such as STL, PLY, OBJ, etc.

[0407] These polygon mesh models can be immediately compatible with and used for subsequent digital dental tasks, such as designing prosthetics using CAD software, fabricating tooth models and surgical guides using 3D printers, and processing final prosthetics (crowns, inlays, etc.) using milling machines.

[0408] In conclusion, this final conversion step plays a crucial role in enabling the scanning system of the present invention to go beyond simply collecting data and become a complete solution that is perfectly integrated into an actual digital dentistry workflow to provide clinically significant results.

[0410] FIG. 4 illustrates a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0411] Although the present disclosure has been described as generally being implementable by a computer device, those skilled in the art will understand that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0412] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will be well aware that the method of the present disclosure may be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computer devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0413] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0414] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0415] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0416] An exemplary environment (1100) for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processor and other multiprocessor architectures may also be used as the processing unit (1104).

[0417] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0418] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0419] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0420] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0421] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0422] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0423] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0424] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means to establish communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0425] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0426] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0427] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0428] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0429] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0430] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0431] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

Claims

Claim 1 An oral scanning system comprising: a multi-axis robotic arm; an oral scanner coupled to the end of the multi-axis robotic arm to acquire three-dimensional data of a scan target; and a control unit that generates and controls the movement of the robotic arm and generates a three-dimensional model by aligning the three-dimensional data acquired from the oral scanner based on the pose of the oral scanner calculated from the kinematic information of the robotic arm; wherein the control unit includes a deep learning-based policy model that determines the next movement of the robotic arm based on data received from the oral scanner, and the policy model includes: a first policy that generates an oral entry movement based on an image from a third-person view camera; and a second policy that generates an oral internal scanning movement based on data from the oral scanner and a three-dimensional model generated so far. Claim 2 An oral scanning system according to claim 1, further comprising a third-person view camera positioned to photograph the robot arm and the scan target, wherein the control unit generates a motion of the robot arm for the oral scanner to enter the oral cavity of the scan target based on an image acquired from the third-person view camera. Claim 3 In paragraph 2, the third-person view camera is an RGBD camera that acquires both color images and depth images, an oral scanning system. Claim 4 In paragraph 2, the oral scanning system, wherein the control unit performs initialization by recognizing the face of the scan target through the third-person view camera at the start of the scanning process and establishing the global coordinate system of the system. Claim 5 In paragraph 4, the above global coordinate system is fixed and set to a specific anatomical feature of the scanned object recognized by the third-person view camera, independently of the base coordinate system of the robot arm, in an oral scanning system. Claim 6 In claim 1, the control unit calculates the pose of the oral scanner by performing a forward kinematic operation based on kinematic information including the joint angles of each of the plurality of joints of the multi-axis robot arm. Claim 7 In claim 6, the pose of the oral scanner is finally calculated using a pre-calculated Hand-Eye transformation matrix that defines the relationship between the transformation matrix of the end coordinate system with respect to the base coordinate system of the multi-axis robot arm and the sensor coordinate system of the oral scanner with respect to the end coordinate system, in an oral scanning system. Claim 8 In claim 1, the control unit finally generates the 3D model by using the pose of the oral scanner calculated from the kinematic information as a reference pose and accumulating and aligning the 3D data sequentially acquired from the oral scanner in a global coordinate system. Claim 9 An oral scanning system according to claim 1, wherein the three-dimensional model is a three-dimensional point cloud or voxel grid accumulated on a global coordinate system. Claim 10 An oral scanning system according to claim 1, further comprising an emergency stop means that can be activated by an operator or a scan target and immediately stops all movements of the multi-axis robotic arm upon activation. Claim 11 delete Claim 12 In claim 1, the oral scanning system, wherein the control unit includes a GPU for accelerating the computation of the deep learning-based policy model. Claim 13 An oral scanning system according to claim 1, wherein the policy model processes multi-modal inputs including images from a third-person viewpoint camera, images from the oral scanner, and a three-dimensional model that is cumulatively updated based on previously acquired data from the oral scanner to generate movements of the robot arm. Claim 14 In claim 1, the oral scanning system, wherein the policy model receives a three-dimensional model that is cumulatively updated based on real-time images acquired from the oral scanner and data previously acquired from the oral scanner, and outputs an operation including joint angle values ​​for each of a plurality of joints of the multi-axis robot arm. Claim 15 In claim 14, the operation output by the policy model is a delta pose including relative displacement and rotation with respect to the current pose of the oral scanner, an oral scanning system. Claim 16 delete Claim 17 In claim 1, the oral scanning system, wherein the first policy performs the oral entry operation by generating a path that avoids collision between the face of the scanned object and the oral scanner using depth information acquired from the third-person viewpoint camera. Claim 18 In claim 1, the second policy is an oral scanning system that receives data from the oral scanner and images acquired from the third-person viewpoint camera together to generate an internal scanning operation. Claim 19 In paragraph 18, the above second policy is an oral scanning system that adjusts the operation of the robot arm by detecting the risk of collision with soft tissue or the movement of the scan target using images from the third-person view camera during an internal scanning operation. Claim 20 In claim 1, the policy model is a single integrated model that receives data from a third-person viewpoint camera and an oral scanner and sequentially generates an oral entry operation and an internal scanning operation, an oral scanning system. Claim 21 In claim 1, the policy model identifies whether there are hole regions within a generated three-dimensional model where data is missing or the point density is below a reference value, determines the termination of the process if the hole region is not identified, and generates an action of moving the oral scanner outside the oral cavity when the termination is determined, an oral scanning system. Claim 22 In claim 21, the policy model identifies hole areas in a generated 3D model where data is missing or density is lower than a reference value before determining the termination of the scanning process, and generates additional scan paths to fill the identified hole areas, an oral scanning system. Claim 23 In paragraph 1, the policy model is an oral scanning system that is learned through reinforcement learning or imitation learning. Claim 24 In paragraph 23, the learning of the policy model is performed in an offline learning manner using a pre-established expert demonstration dataset or scanning dataset, in an oral scanning system. Claim 25 In paragraph 23, the learning of the policy model is performed by applying data augmentation techniques to increase the amount and diversity of collected learning data, in an oral scanning system. Claim 26 In paragraph 23, the above-mentioned imitation learning is an oral scanning system that utilizes a diffusion policy model that models expert behavior as a probability distribution and generates behavior from noise. Claim 27 In paragraph 23, the above-mentioned imitation learning is an oral scanning system that learns using a dataset collected by synchronizing a third-person view camera image at the time of operation, an oral scanner image, and joint angle data of a robot arm operated by an expert, from a scanning operation demonstrated by an expert through remote operation. Claim 28 In claim 27, the above expert data is collected through direct teaching in which an expert directly grasps and moves the multi-axis robotic arm with their hand to demonstrate a scanning path, in an oral scanning system. Claim 29 In claim 27, the control unit generates expert data by synchronizing data streams output from the third-person view camera, the oral scanner, and the joint encoder of the multi-axis robotic arm based on timestamps, in an oral scanning system. Claim 30 In claim 23, the oral scanning system, wherein the reinforcement learning trains the policy model using a reward function designed to provide a positive (+) reward proportional to the amount of valid data added to the 3D model after the unit operation of the robot arm, and to provide a negative (-) reward when scanning an area where data has already been secured in duplicate. Claim 31 delete Claim 32 In claim 30, the above compensation is calculated by updating the three-dimensional model using an accurate pose calculated from the kinematic information of the robot arm, and then quantitatively measuring the difference between the model before the update and the model after the update, in an oral scanning system.

Images