Mixed Reality System for Craniofacial Trauma Image Processing
The mixed reality system addresses the limitations of traditional 2D craniofacial trauma planning by reconstructing and projecting stereoscopic images for precise surgical planning and alignment, improving surgical accuracy and patient understanding.
Patent Information
- Application Number
- TW115200316
- Authority / Receiving Office
- TW · TW
- Patent Type
- Utility models
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2026-07-11
- Estimated Expiration
- 2035-04-29
AI Technical Summary
Traditional surgical planning for craniofacial trauma relies on 2D images, which lack intuitiveness and accuracy, leading to increased surgical errors and patient anxiety due to the difficulty in mentally translating two-dimensional images into three-dimensional anatomical structures, especially in complex fracture repairs where bone fragments may be misaligned or lost.
A mixed reality system comprising an image processing module, stereo imaging module, and sensing control module to reconstruct craniofacial trauma images by dividing them into skin and bone layers, identifying fracture sites, and projecting stereoscopic images for intuitive interaction and adjustment, enabling precise surgical planning and alignment.
Enhances surgical accuracy by providing high-resolution stereoscopic images for precise bone alignment, reduces patient anxiety through intuitive surgical plan understanding, and facilitates medical training and education.
Smart Images
Figure IMG-2_DRAW_115200316-A0305-14-0001-1 
Figure IMG-2_DRAW_115200316-A0305-14-0002-2 
Figure IMG-2_DRAW_115200316-A0305-14-0003-3
Abstract
Description
Mixed Reality System for Craniofacial Trauma Image Processing Technical Field
[0001] This invention relates to a mixed reality system, particularly a mixed reality system for processing craniofacial trauma images. Prior Technology
[0002] In plastic surgery and facial fracture repair, preoperative planning and intraoperative procedures have a decisive impact on the success rate of the surgery and the patient's postoperative satisfaction. However, traditional surgical planning still has many shortcomings.
[0003] Currently, doctors mainly rely on paper or black-and-white 2D images for preoperative planning. This method not only lacks intuitiveness, but patients often find it difficult to accurately understand the surgical process and expected results, easily leading to anxiety and unease. In addition, during the surgery, doctors must rely on personal experience to determine the fracture reduction and implant placement, lacking immediate and intuitive reference information. This may increase surgical errors and affect the final repair outcome.
[0004] Traditional surgical aids primarily include preoperative X-rays, computed tomography (CT) scans, or MRI images. However, these images are mostly presented in two dimensions, requiring surgeons to mentally translate them into three-dimensional anatomical structures. This presents a significant challenge for complex fracture repair surgeries. Especially in facial fracture surgery, the complex bone structure means bone fragments may be misaligned, shifted, or lost. Accurate repositioning and ensuring symmetry are crucial for surgical success. Furthermore, surgeons cannot immediately confirm the location of blood vessels, nerves, and soft tissues during the procedure, increasing surgical risks and potentially leading to complications.
[0005] Therefore, how to develop a mixed reality system for processing craniofacial trauma images to solve the above problems has become a problem to be solved in the relevant technical field. Summary of the Invention
[0006] To address the aforementioned problems, according to one embodiment, this invention provides a mixed reality system for craniofacial trauma image processing, comprising multiple hardware modules consisting of multiple hardware circuits, including an image processing module, a stereo imaging module, a stereo projection module, and a sensing and control module.
[0007] The aforementioned image processing module is communicatively connected to a hospital host computer. After receiving a patient's skull medical image transmitted from the host computer, the module divides the image into skin layer and bone layer images. Based on normal suture information, the module compares the bone layer image to identify abnormal sutures in the skull, generating a fracture site area. Then, it takes at least three symmetrical points between the frontal view of the skull image and the non-fracture site area information to generate a central plane image. Finally, it performs mirroring based on the central plane image and the non-fracture site area information to continuously stitch together important bone fragment information from the irregular edge region outside the fracture site area information, generating a reconstructed skull image. The aforementioned stereoscopic imaging module is communicatively connected to the image processing module. It receives the reconstructed skull image transmitted by the image processing module, reconstructs, and outputs a reconstructed skull stereoscopic image. The aforementioned stereoscopic projection module is communicatively connected to the stereoscopic imaging module. After receiving the reconstructed skull stereoscopic image, it projects the image onto a user-specified device or target area. The aforementioned sensing control module is communicatively connected to the aforementioned stereoscopic projection module to sense an operation command from the user and adjust the reconstructed three-dimensional image of the skull according to the operation command.
[0008] According to this embodiment, preferably, when the information of the non-fracture site area and the information of the fracture site area do not have symmetrical mirroring conditions, the image processing module further splices the primary bone block information continuously from the irregular edge area outside the information of the fracture site area based on a normal skull information to generate the reconstructed skull information.
[0009] According to this embodiment, preferably, the above-mentioned at least three symmetrical points are located on the central axis of the skull.
[0010] According to this embodiment, preferably, the above-mentioned medical images of the patient's skull are obtained by computed tomography or magnetic resonance imaging.
[0011] According to this embodiment, preferably, the above-mentioned sensing control module and the above-mentioned stereoscopic projection module are disposed on a mixed reality glasses device.
[0012] According to this embodiment, preferably, the above-mentioned mixed reality system for craniofacial trauma image processing further includes a stereo printing module, which is communicatively connected to the image processing module, for printing the corresponding primary bone block structure model after receiving the secondary bone block information transmitted by the image processing module.
[0013] The claimed benefits of this invention include: (1) reducing surgical errors and improving the accuracy of bone alignment through stereoscopic image projection; (2) providing high-resolution stereoscopic images, enabling physicians to plan surgical steps and bone repositioning strategies more precisely; (3) allowing patients and their families to intuitively understand the surgical plan through stereoscopic images, reducing preoperative anxiety and increasing trust in medical decisions; and (4) enabling medical students and residents to conduct surgical simulation training in a virtual environment, improving surgical skills and clinical judgment. Simple Explanation of the Diagram
[0014] To make the above-described technology and other objects, features, advantages and embodiments of this invention more apparent and understandable, the accompanying drawings are described below: Figure 1 is a schematic diagram of the architecture of a mixed reality system for processing craniofacial trauma images according to one embodiment of the present invention. Figure 2 is a schematic diagram of a three-dimensional image of the skull without reconstruction according to one embodiment of the present invention. Figure 3 is a schematic diagram of a reconstructed three-dimensional image of the skull according to one embodiment of the present invention. Figure 4 is a schematic diagram illustrating a doctor using a mixed reality system to interpret a patient's condition according to one embodiment of the present invention. Figure 5 is a flowchart illustrating a mixed reality system for processing craniofacial trauma images according to one embodiment of the present invention. Figure 6 is a schematic diagram of an unreconstructed three-dimensional image of a patient's skull with missing bone fragments according to one embodiment of the present invention. Figure 7 is a schematic diagram of a reconstructed three-dimensional image of a patient's skull with missing bone fragments according to one embodiment of the present invention. Implementation
[0015] To illustrate the various embodiments of this invention in more detail, the following description is provided with reference to the accompanying drawings. It should be understood that when an element is referred to as "connected" or "disposed" on another element, it may indicate that the element is directly located on the other element, or that there may also be an intermediate element connecting the element to the other element. Conversely, when an element is referred to as "directly on another element" or "directly connected to another element," it is understood that this explicitly defines the absence of an intermediate element.
[0016] Please refer to Figure 1. According to one embodiment, this invention provides a mixed reality system 100 for craniofacial trauma image processing. Specifically, it utilizes CT, MRI, or ultrasound to acquire medical images of the patient's craniofacial region. An image processing module 110 identifies the fracture area, stitches together large bone blocks based on the continuity of the fracture edges, and ultimately generates a high-precision reconstructed skull image. This reconstructed three-dimensional image is then projected to facilitate preoperative planning by physicians and to help patients and their families intuitively understand the condition. Furthermore, this invention can accurately map virtual images onto the patient's body surface during surgery, enabling physicians to precisely align the images during the procedure and improve the success rate of fracture reduction.
[0017] Referring again to Figure 1, the mixed reality system 100 for craniofacial trauma image processing consists of multiple hardware modules composed of multiple hardware circuits, including an image processing module 110, a stereo imaging module 120, a stereo projection module 130, a sensing control module 140, and a stereo printing module 150.
[0018] Please refer to Figures 2 and 3. Figure 2 is a schematic diagram of a three-dimensional image of the skull without reconstruction according to one embodiment of the present invention. Figure 3 is a schematic diagram of a three-dimensional image of the skull reconstructed according to one embodiment of the present invention. The image processing module 110 is communicatively connected to the hospital host 160 to receive the patient's skull medical images transmitted by the hospital host 160. The patient's skull medical images can be computed tomography (CT) images or magnetic resonance imaging (MRI) images, providing high-resolution visual information of the skull structure as a basis for surgical planning and evaluation.
[0019] Upon receiving the aforementioned medical images of the patient's skull, the image processing module 110 first performs image preprocessing, including image noise reduction, contrast enhancement, and geometric correction, to improve image quality and make subsequent analysis more accurate. Next, the image processing module 110 uses image segmentation technology to analyze and differentiate the patient's skull medical images into skin layer images and bone layer images. This step can utilize thresholding, region growing, machine learning, or deep learning algorithms to automatically identify different tissue layers to obtain a clear skull outline.
[0020] For the bone layer images, the image processing module 110 further compares the bone layer images with normal suture information 112 to determine abnormal sutures in the patient's skull, thereby generating fracture site information 116. This process can determine whether it is a fracture site by comparing the morphological characteristics of the sutures in the current image with the normal skull anatomy. For example, it can analyze the changes in suture width, edge continuity, and density to identify irregular cracks, ensuring the accuracy of the diagnosis. Or, if a suture that should normally appear is located in the first block, but the corresponding bone fragment in the patient is not within the range and boundary of the first block, it can be determined that the bone fragment is a fracture site.
[0021] Subsequently, the image processing module 110 selects at least three symmetrical points 114, such as the glabella, philtrum, and tip of the nose, located on the central axis 118 of the skull, as reference points based on the frontal and non-fracture area information of the patient's skull medical image, to determine the central plane of the skull. This central plane can be accurately located through anatomical feature analysis and image processing algorithms to ensure the accuracy and consistency of the subsequent endoscopic restoration process.
[0022] Based on the generated central plane information, the image processing module performs a mirroring operation, mirroring the non-fractured areas. This mirroring process aims to start from the irregular outer edge of the fractured area, continuously stitching and filling in the missing skull areas to generate a complete and reconstructed skull image.
[0023] For example, suppose a patient suffers a large fracture of the left frontal bone due to trauma, while the right frontal bone remains intact. The image processing module 110 first distinguishes the bone layer images using image segmentation technology and uses normal suture information to determine the fracture area of the left frontal bone. Next, the image processing module 110 selects symmetrical points 114 on three central axes 118, such as the glabella, philtrum, and tip of the nose, to determine the central plane of the skull. In the mirror completion stage, the image processing module 110 uses the right frontal bone as a reference and performs reflection mirroring to project its structure onto the fracture area of the left frontal bone. This allows for the continuous stitching of a key bone fragment from the irregular edge region outside the fracture site information, generating a reconstructed skull information. The key bone fragment information refers to large bone blocks, ensuring that the reconstructed skull structure is complete and conforms to the patient's anatomical characteristics. This process generates a reconstructed skull information.
[0024] 6 and 7, plotted in Figure 6 are a schematic diagram of an unreconstructed stereoscopic image of the cranial bone in the presence of a bony mass missing from the cranium of a patient according to one of the embodiments of the present novel. 7 shows a schematic diagram of a reconstructed stereoscopic image of the cranial bone in the presence of a bony block deletion in the cranium of a patient according to one embodiment of the present novel. Figure 6 Due to the presence of a missing bone mass on the left side of the patient's cranium, it was impossible to completely use the three points for central plane symmetry, so the contralateral image that the right wound measurement should be acceptable was filled with a repair. When there is a missing bone block in the patient's cranium, i.e., the symmetrical mirroring conditions cannot be satisfied in the non-fracture site area and the fracture site area, the image processing module 110 contiguously splices important bone block messages from the irregular marginal region based on the default normal cranial messages (i.e., cranial standard data for healthy populations, representing no abnormal cranial structures). These spliced bone block messages will further complement and reconstruct the missing cranial parts, thereby generating a reconstructed cranial message that is more complete and conforms to the human anatomy, as shown in Figure 7 . This step significantly improves the accuracy of craniofacial trauma image processing and enables effective image reconstruction when symmetrical mirroring conditions cannot be found, ensuring that patients receive high-quality medical image support.
[0025] In addition, the image processing module 110 can also correct the details of the complement area by morphological filtering so that the edges are smooth and consistent with the anatomical features of a normal cranium. Ultimately, after the above processing, the image processing module 110 generates the reconstructed cranial message of the patient.
[0026] The stereo imaging module 120 communicates connected to the image processing module 110 for receiving a reconstructed cranial message transmitted by the image processing module 110, reconstructing and outputting a reconstructed cranial stereo image.
[0027] The stereoscopic projection module 130 communicates to connect the stereoscopic imaging module 120 to project a reconstructed stereoscopic image of the cranial bone, such as an operating room wall, tabletop, or patient's body surface, at a device or target area specified by a user after receiving a reconstructed stereoscopic image of the cranial bone, enabling the structure and details of the surgical site to be presented in an intuitive manner.
[0028] The sensing control module 140 communicates connected to the stereo projection module 130 , which is responsible for sensing the user’s operating commands and adjusting the reconstructed stereoscopic images of the cranial bones accordingly to provide the healthcare provider with a more flexible and intuitive way of interacting with images. The sensing control module 140 and the stereo projection module 130 are jointly disposed on a hybrid reality spectacle device, enabling the healthcare provider to perform real-time manipulation via the wearable device, enhancing convenience during preoperative planning, explanation, and surgical procedures.
[0029] The sensing control module 140 can capture user commands through built-in sensors (such as a gyroscope, accelerometer, and depth camera). These commands may include:
[0030] 1. Head movements: Controlling the field of view by turning or nodding the head, such as rotating a skull image to observe the structure from different angles.
[0031] 2. Hand gestures: Operate by opening your palm to enlarge the image, pinching your fingers to shrink the image, or swiping to switch the image perspective.
[0032] 3. Combination of head and hand movements: For example, use gestures to select an image area and use head movements to fine-tune the image position to improve the precision and intuition of the operation.
[0033] Through the real-time adjustment function of the sensor control module 140, medical personnel can examine the skull structure, fracture sites and repair areas in detail, and adjust the image viewing angle, zoom level or rotation direction as needed to obtain the best viewing angle.
[0034] When a patient has missing bones and the physician determines that 3D printing is needed to manufacture a replacement for the missing bones, the stereo printing module 150 receives the secondary bone block information transmitted by the image processing module and prints the corresponding primary bone block structure model accordingly.
[0035] Example 1
[0036] Please refer to Figure 4, which is a schematic diagram of a doctor using a mixed reality system to interpret a patient's condition according to one embodiment of the present invention.
[0037] Suppose a patient suffers multiple skull fractures due to craniofacial trauma, including large-area bone fissures and the loss of some thin bone fragments. To assist the medical team in planning the surgery and improving surgical precision, the hybrid reality system 100 for craniofacial trauma image processing described in this novel is used to generate and adjust reconstructed three-dimensional images of the skull.
[0038] The patient undergoes computed tomography (CT) and magnetic resonance imaging (MRI) scans to obtain complete information on the skull structure, including the location of fractures, the extent of bone cracks, and the extent of soft tissue damage. First, the image processing module 110 separates the images into skin layer images and bone layer images, so that medical personnel can independently examine the structural features of different tissues.
[0039] Next, the image processing module 110 performs in-depth analysis of the bone layer, detecting abnormal sutures through image recognition technology and marking them as fracture sites. For the fracture area, the image processing module 110 automatically stitches together the fracture edges to reconstruct large bone blocks, ensuring the basic structural integrity of the skull.
[0040] The image processing module 110 applies the principle of facial symmetry, selects at least three symmetrical points to determine the central plane, and fills in the missing thin bone blocks through mirroring technology to generate complete reconstructed skull information.
[0041] After the skull image is processed, the reconstructed skull information is transmitted to the stereo imaging module 120 to reconstruct and output a reconstructed skull stereo image.
[0042] The stereoscopic projection module 130 projects a reconstructed stereoscopic image of the skull onto a user-designated area, allowing medical personnel to explain the procedure to the patient's family before surgery using the stereoscopic image. During the explanation, both the doctor and the patient's family wear mixed reality glasses 170. The doctor uses the sensor control module 140 to zoom, rotate, or move the reconstructed stereoscopic image 122 of the skull to demonstrate the procedure. Through this stereoscopic image, the patient and family can intuitively understand the condition and fully comprehend the possible changes after surgery. This visual communication method significantly reduces the fear and anxiety of patients and their families regarding surgery and enhances their trust in the treatment plan.
[0043] Please refer to Figure 5, which is a flowchart of a mixed reality system 100 for craniofacial trauma image processing according to one embodiment of the present invention.
[0044] In step 200, the image processing module 110 receives a medical image of a patient's skull transmitted from the hospital host 160.
[0045] In step 201, the image processing module 110 divides the skull medical image into skin layer image and bone layer image.
[0046] In step 202, the image processing module 110 compares the bone layer image with normal suture information to determine the abnormal sutures of the patient's skull to generate information about the fracture site area.
[0047] In step 203, the image processing module 110 takes at least three symmetrical points from the front and non-fracture areas of the patient's skull medical image and generates a central plane information.
[0048] Because the types of skull fractures vary greatly in clinical practice, some skull injuries still maintain good left-right or front-back symmetry, while others, due to differences in the direction of external force or the extent of fracture, result in a lack of usable symmetrical mirroring conditions for the overall skull structure. Based on this, this novel design incorporates a conditional judgment step in step 204, with different subsequent processing procedures corresponding to the judgment result. In step 204, it checks whether the information from the non-fractured area and the fractured area possesses symmetrical mirroring conditions. If so, it proceeds to step 205.
[0049] In step 205, the image processing module 110 performs mirroring based on the central plane information and the non-fracture area information to continuously stitch together important bone block information from the irregular edge area outside the fracture area information to generate a reconstructed skull information.
[0050] In step 206, the stereo imaging module 120 receives the reconstructed skull information transmitted by the image processing module 110, reconstructs and outputs a reconstructed stereo image of the skull.
[0051] In step 207, the stereoscopic projection module 130 projects a reconstructed stereoscopic image of the skull onto a device or target area specified by the user.
[0052] In step 207A, the sensing control module 140 senses the operator.
[0053] In step 208, does the sensing control module 140 sense an operation command? If yes, proceed to step 209; if no, return to step 207A, whereby the sensing control module 140 senses the operator until the operator has finished using the projected reconstructed skull stereoscopic image.
[0054] In step 209, the sensing control module 140 adjusts the reconstructed three-dimensional image of the skull according to the operation command, and then returns to continue to step 207A, where the sensing control module 140 senses the operator until the operator has finished using the projected reconstructed three-dimensional image of the skull.
[0055] Because the types of skull fractures vary greatly in clinical practice, some skull injuries still maintain good left-right or front-back symmetry, while others, due to differences in the direction of external force or the extent of fracture, result in the overall skull structure lacking the conditions for symmetrical mirroring. Based on this, this novel design incorporates a conditional judgment step in step 204, with different subsequent processing procedures corresponding to the judgment result. In step 204, it checks whether the information from the non-fractured area and the fractured area possesses the conditions for symmetrical mirroring. If not, it proceeds to step 210.
[0056] In step 210, the image processing module 110 continuously stitches together primary bone block information from the irregular edge region outside the fracture site region information based on normal skull information to generate reconstructed skull information, and then continues to step 206.
[0057] In summary, this novel invention combines mixed reality technology to digitize patient medical images and generate a reconstructed 3D image of the skull, providing a more precise surgical aid for craniofacial trauma. Through its image processing module, this invention divides the received medical images into skin and bone layers, further identifying abnormal sutures in the skull as fracture sites, and stitching together large bone blocks based on the continuity of the fracture edges to generate a complete reconstructed 3D image of the skull. Furthermore, when thin bone blocks are missing, this invention can select at least three symmetrical points using facial symmetry technology to determine the central plane and mirror-complete the missing area, making the skull structure more complete. This invention provides the following benefits:
[0058] (1) Improve the accuracy of reconstructed three-dimensional skull images: By performing fine layering processing on the patient's medical images and stitching together large bone blocks according to the continuity of the fracture edges, the three-dimensional skull image is accurately reconstructed. For missing thin bone blocks, mirror-image completion is performed based on facial symmetry to ensure the accuracy of skull reconstruction, thereby providing accurate reference for surgery.
[0059] (2) Improve surgical precision: By presenting the patient's skull structure and fracture site in the form of a reconstructed three-dimensional image of the skull, medical personnel can make detailed plans before surgery, thereby improving the accuracy of preoperative diagnosis and the surgical process.
[0060] (3) Effective patient communication and understanding: By presenting skull reconstruction images, patients and their families can more intuitively understand the condition and surgical plan, improve the efficiency of communication between doctors and patients, and reduce patients' anxiety.
[0061] (4) Medical Education and Training: The mixed reality system provided by this new system has the advantage of high visualization. By displaying the reconstructed three-dimensional image of the skull in the real world in a three-dimensional form, it can help medical students and clinicians to learn and train intuitively. It allows trainees to practice surgical operations without involving real patients, thereby improving their surgical skills and understanding of complex cases.
[0062] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0063] The above-mentioned implementation examples are merely illustrative examples and are not intended to limit the scope of this invention. Any equivalent modifications or alterations made to them shall not depart from the spirit and scope of this invention and shall be included in the scope of this patent application.
[0064] 100: Mixed Reality System for Craniofacial Trauma Image Processing 110: Image Processing Module 112: Normal suture information 114: Symmetrical point 116: Information on the fracture site area 118: Central axis 120: Stereo Imaging Module 122: Reconstructed 3D Skull Image 130: Stereoscopic projection module 140: Sensor Control Module 150: 3D Printing Module 160: Hospital Mainframe 170: Mixed Reality Glasses Device 200~210, 207A: Steps
Claims
1. A mixed reality system for processing craniofacial trauma images, comprising a plurality of hardware modules consisting of a plurality of hardware circuits, the hardware modules including: An image processing module, communicatively connected to a hospital host, receives a patient's skull medical image transmitted by the host. It then divides the image into skin and bone layers. Based on normal suture information, the module compares the bone layer image to identify abnormal sutures in the skull, generating a fracture site area. It then takes at least three symmetrical points between the front view of the skull image and the non-fracture site area information to generate a central plane image. Finally, it performs mirroring based on the central plane image and the non-fracture site area information to continuously stitch together important bone fragment information from the irregular edge region outside the fracture site area information, generating a reconstructed skull image. A stereoscopic imaging module, communicatively connected to the image processing module, receives the reconstructed skull image transmitted by the image processing module, reconstructs, and outputs a reconstructed skull stereoscopic image. A stereoscopic projection module, communicatively connected to the stereoscopic imaging module, projects the reconstructed skull stereoscopic image onto a user-specified device or target area after receiving the image. A sensing control module is connected to the stereoscopic projection module to sense a user's operation command and adjust the reconstructed stereoscopic image of the skull according to the operation command.
2. The mixed reality system for craniofacial trauma image processing as described in claim 1, wherein when the information of the non-fracture site region and the information of the fracture site region do not have symmetrical mirroring conditions, the image processing module further stitches together primary bone block information from the irregular edge region outside the information of the fracture site region based on normal skull information to generate the reconstructed skull information.
3. A mixed reality system for processing craniofacial trauma images as described in claim 1, wherein the at least three symmetrical points are located on the central axis of the skull.
4. A mixed reality system for processing craniofacial trauma images as described in claim 1, wherein the medical images of the patient's skull are acquired by computed tomography or magnetic resonance imaging.
5. The mixed reality system for craniofacial trauma image processing as described in claim 1, wherein the sensing control module and the stereoscopic projection module are mounted on a mixed reality glasses device.
6. The mixed reality system for craniofacial trauma image processing as described in claim 1 further includes a stereo printing module, communicatively connected to the image processing module, for printing the corresponding primary bone block structure model upon receiving primary bone block information transmitted by the image processing module.