3D medical image model alignment system and method based on computer vision
Patent Information
- Application Number
- TW114109147
- Authority / Receiving Office
- TW · TW
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-03-12
- Publication Date
- 2026-09-16
- Estimated Expiration
- 2045-03-11
AI Technical Summary
Existing medical imaging technologies face challenges in presenting three-dimensional medical images in a cost-effective manner, leading to difficulties in doctor-patient communication and increased surgical risks due to the lack of clear visualization of lesion locations.
A computer vision-based system that aligns a three-dimensional medical image model with a two-dimensional image using face recognition and Delaunay triangulation to construct triangular meshes, establish correspondences, and calculate a pose adjustment matrix for superimposition.
Improves doctor-patient communication and reduces surgical risks by providing clear visual correlation of medical images with actual physical contours.
Smart Images
Figure TWG2TA001075851_001 
Figure TWG2TA001075851_002 
Figure TWG2TA001075851_003
Abstract
Description
[Technical Field]
[0001] This invention relates to an image alignment and synthesis system and method, particularly an image alignment and synthesis system and method for aligning and synthesizing a three-dimensional medical image model with a two-dimensional image. [Previous Technology]
[0002] Medical imaging technology has evolved to include a variety of applications such as X-rays, ultrasound, gamma rays, and nuclear magnetic resonance imaging.
[0003] For example, by integrating technologies that correspond to different spatial locations in medical images, accurate and information-rich multimodal medical images can be provided. Furthermore, multimodal medical images have wide applications in 3D medical imaging. For instance, combining soft tissue details from MRI images with skeletal structure information from CT images can be used for preoperative planning, or to precisely define the treatment area in tumor localization and radiotherapy. In addition, 3D multimodal images can also help analyze the correlation between lesions and surrounding tissues, improving diagnostic accuracy and treatment efficiency. However, in recent years, the use of 3D medical imaging has also faced some challenges. For example, existing surgical navigation systems or augmented reality (AR) equipment are expensive and difficult to popularize in clinical practice. Using high-cost equipment only for preliminary diagnosis or doctor-patient communication purposes is not cost-effective.
[0004] Because the above-mentioned clinical applications lack a cost-effective method that can intuitively correspond medical images to actual appearance contours, it is impossible to clearly observe the actual location of lesions in patients. Therefore, when making diagnoses and planning treatment, physicians need to rely on more abstract imaging information, which increases the risk of interpretation errors. In terms of doctor-patient communication, patients will find it difficult to intuitively understand their own condition or treatment plan, leading to diagnostic errors or insufficient treatment scope, as well as reduced patient trust and cooperation in treatment, which further affects the treatment effect.
[0005] In summary, it can be seen that prior art has long had the problem of presenting three-dimensional medical images on the outline of images acquired by optical acquisition devices in a cost-effective manner, which leads to difficulties in doctor-patient communication and increases the risk of surgical injury. Therefore, it is necessary to propose improved technical means to solve this problem. [Summary of the Invention]
[0006] In view of the problems in the prior art, which, while cost-effective, allows three-dimensional medical images to be displayed on the outline of images acquired by optical capturing devices, leading to difficulties in doctor-patient communication and increased risks of surgical injury, the present invention discloses a computer vision-based three-dimensional medical image model alignment system and method, wherein:
[0007] This invention discloses a computer vision-based three-dimensional medical image model alignment system and method.
[0008] First, this invention discloses a computer vision-based three-dimensional medical image model alignment system. This system includes: a medical image synthesis device, which further includes: a non-transitory computer-readable storage medium and a hardware processor. The non-transitory computer-readable storage medium stores a plurality of computer-readable instructions; and the hardware processor is electrically connected to the non-transitory computer-readable storage medium, executing the computer-readable instructions to cause the medical image synthesis device to perform the following steps:
[0009] Receive a 3D medical image model; use a face recognition machine learning model to identify the model face contour and multiple model face feature points of the 3D medical image model; based on the model face contour and multiple model face feature points, use the Delaunay triangulation algorithm to establish a model face triangular mesh with a fixed number of grid points for the 3D medical image model, the grid coordinates of the model face triangular mesh including X coordinates, Y coordinates and depth values; receive a face image containing a single face; use a face recognition machine learning model to identify the image face contour and multiple image face feature points of the face image; based on the image face contour and multiple image face feature points, use the Delaunay triangulation algorithm to establish an image face triangular mesh with a fixed number of grid points for the face image, the grid coordinates of the image face triangular mesh including X coordinates and Y coordinates, wherein the image face triangular mesh... The number of meshes is the same as the number of meshes in the model's face triangle mesh. A correspondence is established between each model face feature point and each image face feature point. Then, a correspondence is established between the model's face triangle mesh points, sequentially connected to the model's face feature points, and the image's face triangle mesh points, sequentially connected to the image's face feature points. A pose adjustment matrix is calculated for each model face triangle mesh point and its corresponding image face triangle mesh point to adjust the pose of the 3D medical image model to that of the face image. After adjusting the pose of the 3D medical image model using the pose adjustment matrix, the adjusted 3D medical image model is superimposed onto the face image to generate a synthetic medical face image.
[0010] In addition, the present invention discloses a method for aligning three-dimensional medical image models based on computer vision, which includes the following steps:
[0011] First, the medical image synthesis device receives a three-dimensional medical image model; then, the medical image synthesis device uses a face recognition machine learning model to identify the model face contour and multiple model face feature points of the three-dimensional medical image model; next, based on the model face contour and multiple model face feature points, the medical image synthesis device uses the Delaunay triangulation algorithm to construct a model face triangular mesh with a fixed number of grid points from the three-dimensional medical image model, the grid coordinates of the model face triangular mesh including X coordinates, Y coordinates and depth values; next, the medical image synthesis device receives a face image containing a single face; next, the medical image synthesis device uses a face recognition machine learning model to identify the image face contour and multiple image face feature points of the face image; next, based on the image face contour and multiple image face feature points, the medical image synthesis device uses the Delaunay triangulation algorithm to construct a model face triangular mesh with a fixed number of grid points from the three-dimensional medical image model. A triangular mesh for the face in the image is established with a fixed number of grid points. The grid coordinates of the triangular mesh for the face in the image include X and Y coordinates, and the number of grid points in the triangular mesh for the face in the image is the same as the number of grid points in the triangular mesh for the face in the model. Next, the medical image synthesis device establishes a correspondence between each feature point of the face in the model and each feature point of the face in the image, and then establishes a correspondence between the triangular mesh points of the face in the model and the triangular mesh points of the face in the image, which are sequentially connected to the feature points of the face in the model. Next, the medical image synthesis device calculates a pose adjustment matrix for the 3D medical image model to be pose-adjusted to the face image by each triangular mesh point of the face in the model and each corresponding triangular mesh point of the face in the image. Finally, the medical image synthesis device uses the pose adjustment matrix to adjust the pose of the 3D medical image model, and then overlays the adjusted 3D medical image model onto the face image to generate a synthetic medical face image.
[0012] The system and method disclosed in this invention are as described above. The medical image synthesis device establishes a model face triangular mesh of a three-dimensional medical image model and an image face triangular mesh of a face image. It establishes a correspondence between each model face feature point and each image face feature point. After establishing a correspondence between the model face triangular mesh points connected in sequence with the model face feature points and the image face triangular mesh points connected in sequence with the image face feature points, it calculates the pose adjustment matrix of the three-dimensional medical image model to the face image. After adjusting the pose of the three-dimensional medical image model using the pose adjustment matrix, it overlays the adjusted three-dimensional medical image model onto the face image to generate a medical face synthesis image.
[0013] Through the above-mentioned technical means, the present invention can achieve the technical effects of improving doctor-patient communication and reducing surgical risks.
Implementation Method
[0014] The embodiments of the present invention will be described in detail below with reference to the drawings and examples, so that the implementation process of how the present invention uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.
[0015] The following will first describe the computer vision-based three-dimensional medical image model alignment system disclosed in this invention, and please refer to "Figure 1", which is a system block diagram of the computer vision-based three-dimensional medical image model alignment system of this invention.
[0016] First, the present invention discloses a computer vision-based three-dimensional medical image model alignment system. This system includes: a medical image synthesis device 10, which further includes: a non-transitory computer-readable storage medium 11 and a hardware processor 12. The non-transitory computer-readable storage medium 11 stores a plurality of computer-readable instructions; and the hardware processor 12 is electrically connected to the non-transitory computer-readable storage medium 11 and executes the computer-readable instructions to cause the medical image synthesis device 10 to perform the following steps:
[0017] The medical image synthesis device 10 receives a three-dimensional medical image model 31 from the three-dimensional medical image model providing device 20. For a schematic diagram of the three-dimensional medical image model 31, please refer to "Figure 2A". "Figure 2A" is a schematic diagram of the three-dimensional medical image model alignment based on computer vision according to the present invention. The aforementioned three-dimensional medical image model providing device 20 is, for example, a medical magnetic resonance imaging (MRI) machine, a medical computed tomography (CT) scan machine, a cloud server, a USB device, an SD memory card, etc. It is only an example for illustration and does not limit the application scope of the present invention.
[0018] It is worth noting that in "Figure 2A", a three-dimensional medical image model 31 with the outline of the human body is used as an illustration. The three-dimensional medical image model 31 can also be presented with internal tissues, or the three-dimensional medical image model 31 can be presented with part of the outline of the human body and part of the internal tissues. The three-dimensional medical image model 31 can be generated by a medical magnetic resonance imaging machine or by a medical computed tomography scanner. This is only an example and is not intended to limit the scope of application of the present invention.
[0019] Next, a face recognition machine learning model is used to identify the face contour and face feature points of the three-dimensional medical image model 31, so as to identify the model face contour and multiple model face feature points of the three-dimensional medical image model 31. The aforementioned face recognition machine learning model includes, but is not limited to, MTCNN, RetinaFace, YOLO, Dlib, MediaPipe Face Mesh, Face Alignment Network (FAN), etc. In addition to using a face recognition machine learning model to identify the face contour and face feature points, edge detection technology can also be used to identify the face contour. The aforementioned face feature points include, but are not limited to, pupils, corners of eyes, tip of nose, wing of nose, corners of mouth, etc. This is only an example and is not intended to limit the application scope of the present invention.
[0020] Next, based on the model face contour and multiple model face feature points, the Delaunay triangulation algorithm is used to establish a model face triangular mesh 32 with a fixed number of grid points for the 3D medical image model. The grid coordinates of the model face triangular mesh 32 include X coordinates, Y coordinates, and depth values. It is worth noting that the grid coordinates of the model face triangular mesh 32 can be directly the 3D coordinate information of the 3D medical image model 31, or the 3D coordinate information of the 3D medical image model 31 can be back-projected to the X coordinates, Y coordinates, and depth values in a 2D coordinate system. This is only an example and does not limit the application scope of the present invention. For a schematic diagram of the model face triangular mesh 32, please refer to "Figure 2B". "Figure 2B" shows a schematic diagram of the model face triangular mesh for the alignment of the 3D medical image model based on computer vision of the present invention. Specifically, if the face recognition machine learning model is Dlib, the aforementioned fixed number of grid points will be 68. If the face recognition machine learning model is MediaPipe Face... The Mesh, as mentioned above, has a fixed number of 468 grid points. This is merely an example and is not intended to limit the scope of application of the present invention.
[0021] The medical image synthesis device 10 receives a face image 41 containing a single face from an external optical capturing device 40. For a schematic diagram of the face image 41, please refer to "Figure 3A". "Figure 3A" is a schematic diagram of the face image aligned with the computer vision-based three-dimensional medical image model of the present invention. Then, a face recognition machine learning model is used to identify the face contour and face feature points of the face image 41, so as to identify the image face contour and multiple image face feature points of the face image 41. The aforementioned optical capturing device 40 is, for example, an optical camera, a monocular or binocular camera, etc., which is only an example and is not intended to limit the application scope of the present invention.
[0022] Next, the medical image synthesis device 10 uses the image face contour and multiple image face feature points as a reference, and uses the Delaunay triangulation algorithm to establish an image face triangular mesh 42 with a fixed number of grid points from the face image 41. The grid coordinates of the image face triangular mesh 42 include X coordinates and Y coordinates. The number of grids in the image face triangular mesh 42 is the same as the number of grids in the model face triangular mesh 32. For a schematic diagram of the image face triangular mesh 42, please refer to "Figure 3B". "Figure 3B" shows a schematic diagram of the image face triangular mesh for the alignment of the three-dimensional medical image model based on computer vision of the present invention.
[0023] In addition, the medical image synthesis device 10 uses the image face contour and multiple image face feature points as a reference, and then matches the model face contour with the image face contour and matches the model face feature points with the image face feature points. Then, it integrates the grid points of the non-model face feature points of the model face triangular mesh 32 with the image face feature points to form the image face triangular mesh 42. This is only an example and is not intended to limit the application scope of the present invention.
[0024] Next, the medical image synthesis device 10 establishes a correspondence between each model facial feature point and each image facial feature point. Specifically, the medical image synthesis device 10 establishes a correspondence between the left corner of the model facial feature point and the left corner of the image facial feature point; the medical image synthesis device 10 establishes a correspondence between the left right corner of the model facial feature point and the left right corner of the image facial feature point; the medical image synthesis device 10 establishes a correspondence between the right left corner of the model facial feature point and the right left corner of the image facial feature point; the medical image synthesis device 10 establishes a correspondence between the right right corner of the model facial feature point and the right right corner of the image facial feature point. This is only an example and is not intended to limit the scope of application of the present invention.
[0025] The medical image synthesis device 10 further establishes the correspondence between the model face triangular mesh points sequentially connected to the model face feature points and the image face triangular mesh points sequentially connected to the image face feature points. Specifically, please refer to both "Figure 4A" and "Figure 4B". "Figure 4A" shows a schematic diagram of the model feature points and model triangular mesh points aligned in the computer vision-based three-dimensional medical image model of the present invention. "Figure 4B" shows the image feature points and image triangular mesh points aligned in the computer vision-based three-dimensional medical image model of the present invention. The diagram illustrates how the left corner of the left eye of model facial feature point 511 is correlated with the left corner of the left eye of image facial feature point 521; the left eye contour of the adjacent left corner of the left eye of model facial feature point 512 is correlated with the left eye contour of the adjacent right corner of the left eye of image facial feature point 522; the left eye contour of the adjacent left corner of the left eye of model facial feature point 513 is correlated with the left eye contour of the adjacent right corner of the left eye of image facial feature point 523, and so on. Then, the first model face triangle mesh point 531 connected to model facial feature point 511 is... A correspondence is established between the first image face triangle mesh point 541 connected to the image face feature point 521, the second model face triangle mesh point 532 connected to the first model face triangle mesh point 531, and the second image face triangle mesh point 542 connected to the first image face triangle mesh point 541, and so on. This is only an example and is not intended to limit the application scope of the invention. It is worth noting that the model face triangle mesh points sequentially connected to the model face feature points are those using model face feature points (e.g., model face feature points). The grid point order for establishing the model face triangle mesh using the Idelaunay triangulation algorithm with face feature point 511 and model face feature point 512 as the starting point; the image face triangle mesh points connected sequentially to the image face feature points are the grid point order for establishing the image face triangle mesh using the image face feature points (e.g., image face feature point 521 and image face feature point 522) corresponding to the model face feature points (e.g., model face feature point 511 and model face feature point 512) as the starting point.
[0026] Next, the medical image synthesis device 10 calculates the pose adjustment matrix of the three-dimensional medical image model 31 to the face image 41 by using each model face triangle mesh point and the corresponding image face triangle mesh point. The pose adjustment matrix is obtained by using, for example, the Perspective-n-Points (PnP) problem, the Random Sample Consistency Perspective-n-Points (RANSAC PnP) method, and the Simultaneous Localization and Mapping (SLAM) algorithm, etc. (These are just examples and do not limit the application scope of the present invention). The aforementioned pose adjustment is either translational pose adjustment or rotational pose adjustment, and the aforementioned pose adjustment matrix includes both translational pose adjustment matrix and rotational pose adjustment matrix.
[0027] Next, the medical image synthesis device 10 uses a pose adjustment matrix to adjust the pose of the three-dimensional medical image model 31, and then superimposes the adjusted three-dimensional medical image model 31 onto the face image 41 to generate a medical face synthesis image 61. For a schematic diagram of the medical face synthesis image 61, please refer to "Figure 5". "Figure 5" is a schematic diagram of the face image of the three-dimensional medical image model based on computer vision of the present invention. It is worth noting that in "Figure 5", the three-dimensional medical image model 31 of internal tissue is used as a schematic diagram. This is only for illustration and the present invention is not limited thereto.
[0028] It should be noted in particular that, in practical implementation, the invention may be implemented partially or completely based on hardware, for example, one or more components in the system may be implemented via an integrated circuit chip, System on Chip (SoC), Complex Programmable Logic Device (CPLD), Field Programmable Logic Device (CPLD), Field Programmable Logic Array (Field Programmable Process Gate) Array, etc. The non-transient computer-readable storage media of the invention, which uploads computer-readable instructions (or referred to as computer program instructions) for enabling the processor to implement various aspects of the invention, the non-transient computer-readable storage media may be a tangible device that can hold and store instructions for use by the command execution equipment. Non-transient computer-readable storage media may be, but are not limited to, electrical storage equipment, magnetic storage equipment, optical storage equipment, electromagnetic storage equipment, semiconductor storage equipment, or any suitable combination of the above. More specific examples of computer-readable storage media (not an exhaustive list) include: hard drives, random access memory, read-only memory, flash memory, optical discs, floppy disks, and any suitable combination of the above. The non-transient computer-readable storage media used here are not construed as instantaneous signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated through waveguides or other transmission media (e.g., optical signals via fiber optic cables), or electrical signals transmitted through wires. In addition, the computer-readable commands described here can be downloaded to various computing / processing equipment from non-transient computer-readable storage media, or to external computer equipment or external storage equipment via a network, such as an Internet, LAN, WAN and / or wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, hubs and / or gateways. The network card or network interface in each computing / processing equipment receives computer-readable commands from the network and forwards these computer-readable commands for storage in the non-temporal computer-readable storage media in the respective computing / processing equipment. Computer-readable instructions to perform operations of the present invention may be combination language instructions, instruction set architecture instructions, machine instructions, machine-related instructions, microinstructions, firmware instructions, or source code or object code (Object Code) written in any combination of one or more programming languages, including object-oriented programming languages, such as Common Lisp, Python, C++, Objective-C, Smalltalk, Delphi, Java, Swift, C#, Perl, Ruby and PHP, etc., as well as conventional procedural (Procedural) programming languages, such as C language or similar programming languages.
[0029] Next, the operation method of the present invention will be described below, and please refer to Figures 6A and 6B at the same time. Figures 6A and 6B are flowcharts of the method of the computer vision-based three-dimensional medical image model alignment method of the present invention.
[0030] This invention discloses a method for aligning three-dimensional medical image models based on computer vision, which includes the following steps:
[0031] First, the medical image synthesis device receives a three-dimensional medical image model (step 601); then, the medical image synthesis device uses a face recognition machine learning model to identify the model face contour and multiple model face feature points of the three-dimensional medical image model (step 602); next, based on the model face contour and multiple model face feature points, the medical image synthesis device uses the Delaunay triangulation algorithm to establish a model face triangular mesh with a fixed number of grid points for the three-dimensional medical image model, the grid coordinates of the model face triangular mesh including X coordinates, Y coordinates and depth values (step 603); next, the medical image synthesis device receives a face image containing a single face (step 604); next, the medical image synthesis device uses a face recognition machine learning model to identify the image face contour and multiple image face feature points of the face image (step 605); next, based on the image face contour and multiple image face feature points, the medical image synthesis device uses the Delaunay triangulation algorithm to segment the image face into a triangular mesh containing a single face. A face image triangular mesh with a fixed number of grid points is created from the face image. The grid coordinates of the face image triangular mesh include X and Y coordinates, and the number of grids in the face image triangular mesh is the same as the number of grids in the model face triangular mesh (step 606). Next, the medical image synthesis device establishes a correspondence between each model face feature point and each image face feature point, and then establishes a correspondence between the model face triangular mesh points sequentially connected to the model face feature points and the image face triangular mesh points sequentially connected to the image face feature points (step 607). Next, the medical image synthesis device calculates a pose adjustment matrix for adjusting the pose of the three-dimensional medical image model to the face image by using each model face triangular mesh point and the corresponding image face triangular mesh point (step 608). After adjusting the pose of the three-dimensional medical image model using the pose adjustment matrix, the medical image synthesis device overlays the adjusted three-dimensional medical image model onto the face image to generate a medical face composite image (step 609).
[0032] In summary, the medical image synthesis device establishes a model face triangular mesh for a three-dimensional medical image model, establishes an image face triangular mesh for a face image, establishes a correspondence between each model face feature point and each image face feature point, and then establishes a correspondence between the model face triangular mesh points sequentially connected to the model face feature points and the image face triangular mesh points sequentially connected to the image face feature points. After calculating the pose adjustment matrix of the three-dimensional medical image model to the face image, the pose adjustment matrix is used to adjust the pose of the three-dimensional medical image model, and then the adjusted three-dimensional medical image model is superimposed on the face image to generate a medical face synthesis image.
[0033] This technology can solve the problem that the prior art is cost-effective and can present three-dimensional medical images on the outline of the image acquired by the optical capturing device, which leads to difficulties in doctor-patient communication and increased surgical risk, thereby achieving the technical effect of improving doctor-patient communication and reducing surgical risk.
[0034] Although the embodiments disclosed in this invention are as described above, the content described is not intended to directly limit the scope of patent protection of this invention. Anyone skilled in the art to which this invention pertains may make some modifications in form and detail of the implementation without departing from the spirit and scope disclosed in this invention. The scope of patent protection of this invention shall still be determined by the appended claims. [Simplified Explanation of the Diagram]
[0035] Figure 1 is a system block diagram of the computer vision-based 3D medical image model alignment system of the present invention. Figure 2A is a schematic diagram of the 3D medical image model for computer vision-based 3D medical image model alignment of the present invention. Figure 2B is a schematic diagram of the face triangular mesh of the model for computer vision-based 3D medical image model alignment of the present invention. Figure 3A is a schematic diagram of the face image for computer vision-based 3D medical image model alignment of the present invention. Figure 3B is a schematic diagram of the face triangular mesh of the image for computer vision-based 3D medical image model alignment of the present invention. Figure 4A is a schematic diagram of the model feature points and model triangular mesh points for computer vision-based 3D medical image model alignment of the present invention. Figure 4B is a schematic diagram of the image feature points and image triangular mesh points for computer vision-based 3D medical image model alignment of the present invention. Figure 5 is a schematic diagram of the face image for computer vision-based 3D medical image model alignment of the present invention. Figures 6A and 6B are flowcharts of the computer vision-based 3D medical image model alignment method of the present invention.
Claims
1. A computer vision-based three-dimensional medical image model alignment system, the system comprising: a medical image synthesis device, the medical image synthesis device further comprising: a non-transitory computer-readable storage medium storing a plurality of computer-readable instructions; and a hardware processor electrically connected to the non-transitory computer-readable storage medium, executing the computer-readable instructions to cause the medical image synthesis device to perform the following steps: receiving a three-dimensional medical image model; using a face recognition machine learning model to identify a model face contour and a plurality of model face feature points of the three-dimensional medical image model; using the model face contour and the plurality of model face feature points as a reference, using the Delaunay triangulation algorithm to construct a model face triangular mesh with a fixed number of grid points from the three-dimensional medical image model, the grid coordinates of the model face triangular mesh including X coordinates, Y coordinates and a depth value; receiving a face image containing a single face; using a face recognition machine learning model to identify an image face contour and a plurality of image face feature points of the face image; Based on the image face contour and multiple image face feature points, the Delaunay triangulation algorithm is used to construct an image face triangular mesh with a fixed number of grid points. The grid coordinates of the image face triangular mesh include X and Y coordinates, and the number of grids in the image face triangular mesh is the same as the number of grids in the model face triangular mesh. A correspondence is established between each model face feature point and each image face feature point, and then a correspondence is established between the model face triangular mesh points sequentially connected to the model face feature points and the image face triangular mesh points sequentially connected to the image face feature points. A pose adjustment matrix is calculated for each model face triangular mesh point and each corresponding image face triangular mesh point to adjust the pose of the 3D medical image model to the face image. After adjusting the pose of the 3D medical image model using the pose adjustment matrix, the adjusted 3D medical image model is superimposed onto the face image to generate a medical face composite image.
2. The computer vision-based 3D medical image model alignment system as described in claim 1, wherein the face recognition machine learning model includes MTCNN, RetinaFace, YOLO, Dlib, MediaPipe Face Mesh, and Face Alignment Network (FAN).
3. The computer vision-based three-dimensional medical image model alignment system as described in claim 1, wherein the fixed number of grid points of the model face triangle mesh varies depending on the face recognition machine learning model.
4. The computer vision-based three-dimensional medical image model alignment system as described in claim 1, wherein the fixed number of grid points in the image face triangular mesh varies depending on the face recognition machine learning model.
5. The computer vision-based three-dimensional medical image model alignment system as described in claim 1, wherein the posture adjustment matrix includes a translational posture adjustment matrix and a rotational posture adjustment matrix.
6. A method for aligning a three-dimensional medical image model based on computer vision, comprising the following steps: A medical image synthesis device receives a three-dimensional medical image model; The medical image synthesis device uses a face recognition machine learning model to identify a model face contour and multiple model face feature points of the three-dimensional medical image model; The medical image synthesis device uses the model face contour and multiple model face feature points as a reference, and uses the Delaunay triangulation algorithm to construct a model face triangular mesh with a fixed number of grid points from the three-dimensional medical image model, wherein the grid coordinates of the model face triangular mesh include X coordinates, Y coordinates, and a depth value; The medical image synthesis device receives a face image containing a single face; The medical image synthesis device uses a face recognition machine learning model to identify an image face contour and multiple image face feature points of the face image; The medical image synthesis device uses the facial contour and multiple facial feature points in the image as a reference, and employs the Delaunay triangulation algorithm to construct a facial triangular mesh with a fixed number of grid points from the facial image. The grid coordinates of the facial triangular mesh include X and Y coordinates, and the number of grid points in the facial triangular mesh is the same as the number of grid points in the model facial triangular mesh. The medical image synthesis device establishes a correspondence between each model facial feature point and each image facial feature point, and then establishes a correspondence between the model facial triangular mesh points sequentially connected to the model facial feature points and the image facial triangular mesh points sequentially connected to the image facial feature points. The medical image synthesis device calculates a pose adjustment matrix for adjusting the pose of the three-dimensional medical image model to the facial image using each model facial triangular mesh point and its corresponding image facial triangular mesh point. After adjusting the pose of the three-dimensional medical image model using the pose adjustment matrix, the medical image synthesis device overlays the adjusted three-dimensional medical image model onto the facial image to generate a synthesized medical facial image.
7. The computer vision-based three-dimensional medical image model alignment method as described in claim 6, wherein the face recognition machine learning model includes MTCNN, RetinaFace, YOLO, Dlib, MediaPipe Face Mesh, and Face Alignment Network (FAN).
8. The computer vision-based three-dimensional medical image model alignment method as described in claim 6, wherein the fixed number of grid points of the face triangle mesh in the model is different depending on the face recognition machine learning model.
9. The computer vision-based three-dimensional medical image model alignment method as described in claim 6, wherein the fixed number of grid points of the face triangle mesh in the image is different depending on the face recognition machine learning model.
10. The computer vision-based three-dimensional medical image model alignment method as described in claim 6, wherein the pose adjustment matrix includes a translation pose adjustment matrix and a rotation pose adjustment matrix.