Three-dimensional surface scanning data modeling intelligent interaction fusion method and device and storage medium

Abstract

The invention discloses a three-dimensional surface scanning data modeling intelligent interactive fusion method and device and a storage medium. The method comprises the following steps: collecting a visible light image and a near-infrared light image of the face of a patient at the same time; generating an epidermis three-dimensional grid model based on the visible light image and extracting subcutaneous tissue structure data based on the near-infrared light image; acquiring a multi-angle scanning image of teeth of a patient, and generating a tooth part three-dimensional model; extracting a tooth region contour; segmenting tooth area three-dimensional data matched with the tooth area contour from the tooth part three-dimensional model; and performing data fusion on the three-dimensional mesh model of the epidermis, the subcutaneous tissue structure data and the three-dimensional data of the tooth area to generate a three-dimensional facial anatomical structure model. The method solves the problem that in the prior art, subcutaneous tissue structure information is lacked, and can be used for assisting medical staff in treatment safety assessment and low-risk treatment planning and simulating dynamic change expressions caused by facial muscle movement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for intelligent interactive fusion of three-dimensional surface scan data modeling, electronic devices, and computer-readable storage media. Background Technology

[0002] 3D facial scanning technology plays a crucial role in fields such as cosmetic dentistry and plastic surgery, enabling pre-treatment planning and post-treatment expression simulation. Traditional handheld scanning devices are prone to image quality degradation due to improper operator handling and subject posture changes. Therefore, handheld 3D facial scanning devices have emerged in recent years. These devices utilize multi-angle cameras and structured light projectors to simultaneously capture images of the face and teeth from multiple angles, generating high-quality 3D facial scan data. However, this type of 3D facial scanning technology still has the following limitations: 1) Traditional structured light cannot acquire information about subcutaneous tissue structure such as blood vessel distribution and tissue density, making it difficult to assist medical staff in assessing treatment safety and planning low-risk treatments; 2) During orthodontic treatment, although three-dimensional facial scan data can be used to predict the patient's facial expressions after treatment, the dynamic changes in facial expressions caused by facial muscle movements after orthodontic treatment are ignored (such as the dynamic changes in facial expressions caused by smiling and chewing movements). Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, the present invention aims to provide a three-dimensional facial scan data modeling intelligent interactive fusion method, electronic device, and computer-readable storage medium, which solves the problem of the lack of subcutaneous tissue structure information in existing three-dimensional facial scanning technology. It can be used to assist medical staff in conducting treatment safety assessments and realizing low-risk treatment planning, as well as to simulate dynamic changes in facial expressions caused by facial muscle movements.

[0004] The intelligent interactive fusion method for three-dimensional surface scan data modeling of the present invention is implemented using the following technical solution: A method for intelligent interactive fusion of 3D surface scan data modeling includes the following steps: Simultaneously, visible light and near-infrared images of the patient's face were acquired; A three-dimensional mesh model of the patient's face is generated based on the visible light image, and subcutaneous tissue structure data of the patient's face is extracted based on the near-infrared light image; Acquire multi-angle scan images of the patient's teeth and generate a 3D model of the teeth; Facial feature points are identified from the three-dimensional mesh model of the epidermis using an artificial neural network model, and the contours of the tooth region are extracted. Spatial registration is performed between the three-dimensional model of the tooth part and the contour of the tooth region, and three-dimensional data of the tooth region matching the contour of the tooth region are segmented from the three-dimensional model of the tooth part. Data fusion was performed on the three-dimensional mesh model of the epidermis, subcutaneous tissue structure data, and three-dimensional data of the dental region to generate a three-dimensional facial anatomical model.

[0005] Furthermore, the subcutaneous tissue structure data includes subcutaneous blood vessel distribution data and subcutaneous tissue density data.

[0006] Furthermore, in the step of simultaneously acquiring visible light images and near-infrared light images of the patient's face, the visible light image of the patient's face is acquired using a visible light band camera, and the near-infrared light image of the patient's face is acquired using a near-infrared light band camera. The penetration depth of the near-infrared light band camera is 3~7mm.

[0007] Furthermore, the intelligent interactive fusion method for 3D surface scan data modeling also includes the following steps: A facial muscle kinematics model is used to simulate dynamic facial expressions on a three-dimensional facial anatomical model, and a dynamic prediction sequence after treatment is output.

[0008] Furthermore, the dynamic prediction sequence refers to a three-dimensional data stream of facial deformation that changes over time, used to represent the dynamic changes in facial expressions caused by facial muscle movements after treatment, such as smiling or chewing movements.

[0009] Furthermore, the facial muscles simulated by the facial muscle kinematic model include one or more of the following: orbicularis oculi, orbicularis oris, zygomaticus major, masseter, temporalis, depressor anguli oris, and platysma.

[0010] The electronic device of the present invention is implemented using the following technical solution: An electronic device includes a processor and a memory storing a computer program, wherein when the processor executes the computer program, it implements the above-described intelligent interactive fusion method for three-dimensional surface scan data modeling.

[0011] The computer-readable storage medium of the present invention is implemented using the following technical solution: A computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the above-mentioned intelligent interactive fusion method for three-dimensional surface scan data modeling.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: The three-dimensional facial scan data modeling intelligent interactive fusion method, electronic device, and computer-readable storage medium provided by this invention generate a three-dimensional facial anatomical structure model by fusing data from a three-dimensional epidermal mesh model, subcutaneous tissue structure data, and three-dimensional data of the dental region. On the one hand, this three-dimensional facial anatomical structure model can better assist medical staff in assessing treatment safety and achieving low-risk treatment planning. On the other hand, this three-dimensional facial anatomical structure model can be applied to biomechanical simulation analysis, and then use a facial muscle kinematic model to perform dynamic expression simulation on the three-dimensional facial anatomical structure model. Attached Figure Description

[0013] Figure 1 This is a flowchart of the intelligent interactive fusion method for three-dimensional surface scan data modeling according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0014] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0015] refer to Figures 1-2 This invention provides a method for intelligent interactive fusion of three-dimensional surface scan data modeling, an electronic device, and a computer-readable storage medium. The method includes steps S1 to S7.

[0016] S1. Simultaneously acquire visible light and near-infrared images of the patient's face. Specifically, this can be achieved using a multispectral imaging system. This system may include a visible light camera and a near-infrared camera. The visible light camera acquires visible light images of the patient's face, while the near-infrared camera acquires near-infrared images. The near-infrared camera has a penetration depth of 3-7 mm. Multiple cameras can be configured, similar to existing technologies, corresponding to multiple shooting angles. By simultaneously acquiring visible light and near-infrared images of the patient's face, a perfect match between the facial epidermis and subcutaneous tissue images can be ensured.

[0017] S2. Generate a three-dimensional mesh model of the patient's facial epidermis based on the visible light image, and extract subcutaneous tissue structure data of the patient's face based on the near-infrared light image. The extracted subcutaneous tissue structure data may include subcutaneous blood vessel distribution data and subcutaneous tissue density data.

[0018] S3. Obtain multi-angle scan images of the patient's teeth and generate a 3D model of the teeth. While multi-angle scan images of the patient's teeth can still be obtained using a camera, the images captured by the camera will not be precise images of the tooth area. The original scan images usually include parts surrounding the teeth, such as the chin, nose, and parts of the left and right sides of the face. Therefore, the 3D model of the teeth needs further segmentation.

[0019] S4. Using an artificial neural network model, facial feature points are identified from the three-dimensional mesh model of the epidermis, and the contours of the tooth region are extracted. This artificial neural network model can be a trained convolutional neural network (CNN) model, which has a faster processing efficiency compared to traditional image processing techniques. The CNN model can be trained using multiple sets of medical images labeled with 68 facial feature points and tooth contours (e.g., 50,000 to 100,000 sets) and multiple sets of paired near-infrared-visible light image datasets (e.g., 50,000 to 100,000 sets).

[0020] S5. Spatial registration is performed between the three-dimensional model of the tooth part and the contour of the tooth region, and the three-dimensional data of the tooth region matching the contour of the tooth region is segmented from the three-dimensional model of the tooth part.

[0021] S6. Perform data fusion on the epidermal 3D mesh model, subcutaneous tissue structure data, and dental region 3D data to generate a 3D facial anatomical structure model.

[0022] S7. A dynamic expression simulation is performed on a three-dimensional facial anatomical model using a facial muscle kinematic model, outputting a dynamic prediction sequence after treatment. This dynamic prediction sequence refers to a three-dimensional data stream of facial deformation changing over time, used to represent the dynamic changes in facial expression caused by facial muscle movements after treatment. For example, these facial muscle movements could be smiling or chewing. For instance, if a smiling action lasts 0.5 seconds, the output dynamic prediction sequence would be a three-dimensional data stream of facial deformation from 0 to 0.5 seconds. Existing techniques for simulating post-treatment expressions display static expressions, while the method of this invention displays dynamically changing expressions. Taking a smiling action as an example, figuratively speaking, a dynamically changing expression displays a dynamic image from 0 to 0.5 seconds, while a static expression displays a still image at 0.5 seconds.

[0023] In step S7, the facial muscles simulated by the facial muscle kinematic model include one or more of the following: orbicularis oculi, orbicularis oris, zygomaticus major, masseter, temporalis, depressor anguli oris, and platysma. The orbicularis oculi can be used to simulate closing the eyes, the orbicularis oris can be used to simulate smiling and pursing, and the masseter can be used to simulate chewing, etc. Generally speaking, simulating all the facial muscles mentioned here will yield the best results.

[0024] The intelligent interactive fusion method for three-dimensional facial scan data modeling provided in this invention generates a three-dimensional facial anatomical structure model by fusing data from a three-dimensional epidermal mesh model, subcutaneous tissue structure data, and three-dimensional data of the dental region. On the one hand, this three-dimensional facial anatomical structure model can better assist medical staff in assessing treatment safety and achieving low-risk treatment planning. On the other hand, this three-dimensional facial anatomical structure model can be applied to biomechanical simulation analysis, and then dynamic expression simulation can be performed on the three-dimensional facial anatomical structure model using a facial muscle kinematic model.

[0025] refer to Figure 2 The electronic device of this invention includes a processor 201 and a memory 202 storing a computer program. When the processor 201 executes the computer program, it implements the intelligent interactive fusion method for three-dimensional surface scan data modeling as described in this invention. The processor 201, memory 202, and communication interface 203 all communicate with each other via a communication bus 204.

[0026] In the electronic device of this embodiment, the processor 201 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. The processor 201 can be a microprocessor or any conventional processor. The processor 201 can call programs stored in the memory 202.

[0027] In the electronic device of this embodiment, the memory 202 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. The memory 202 may store programs for implementing the functions corresponding to the methods described above. In one possible implementation, the memory 202 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one computer program required for a system function; the data storage area may store data created during use. Furthermore, the memory 202 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores the operating system and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0028] In the electronic device of this embodiment, the communication interface 203 can be an interface of a communication module for connecting with other devices or systems. It should be noted that... Figure 2The structure shown does not constitute a limitation on the electronic device in the embodiments of the present invention. In practical applications, the electronic device may include more than Figure 2 More or fewer components as shown, or combinations of certain components.

[0029] The computer-readable storage medium of this invention stores a computer program thereon. When executed by a processor, the computer program implements the intelligent interactive fusion method for three-dimensional surface scan data modeling as described in this invention. The computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0030] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for intelligent interactive fusion of three-dimensional surface scan data modeling, characterized in that, Includes the following steps: Simultaneously, visible light and near-infrared images of the patient's face were acquired; A three-dimensional mesh model of the patient's face is generated based on the visible light image, and subcutaneous tissue structure data of the patient's face is extracted based on the near-infrared light image; Acquire multi-angle scan images of the patient's teeth and generate a 3D model of the teeth; Facial feature points are identified from the three-dimensional mesh model of the epidermis using an artificial neural network model, and the contours of the tooth region are extracted. Spatial registration is performed between the three-dimensional model of the tooth part and the contour of the tooth region, and three-dimensional data of the tooth region matching the contour of the tooth region are segmented from the three-dimensional model of the tooth part. Data fusion was performed on the three-dimensional mesh model of the epidermis, subcutaneous tissue structure data, and three-dimensional data of the dental region to generate a three-dimensional facial anatomical model.

2. The intelligent interactive fusion method for three-dimensional surface scan data modeling as described in claim 1, characterized in that, The subcutaneous tissue structure data includes subcutaneous blood vessel distribution data and subcutaneous tissue density data.

3. The intelligent interactive fusion method for three-dimensional surface scan data modeling as described in claim 1, characterized in that, In the step of simultaneously acquiring visible light and near-infrared light images of the patient's face, the visible light image of the patient's face is acquired using a visible light band camera, and the near-infrared light image of the patient's face is acquired using a near-infrared light band camera. The penetration depth of the near-infrared light band camera is 3~7mm.

4. The intelligent interactive fusion method for three-dimensional surface scan data modeling as described in claim 1, characterized in that, The intelligent interactive fusion method for 3D surface scan data modeling also includes the following steps: A facial muscle kinematics model is used to simulate dynamic facial expressions on a three-dimensional facial anatomical model, and a dynamic prediction sequence after treatment is output.

5. The intelligent interactive fusion method for three-dimensional surface scan data modeling as described in claim 4, characterized in that, The dynamic prediction sequence refers to a three-dimensional data stream of facial deformation that changes over time, used to represent the dynamic changes in facial expressions caused by facial muscle movements after treatment, such as smiling or chewing movements.

6. The intelligent interactive fusion method for three-dimensional surface scan data modeling as described in claim 4, characterized in that, The facial muscles simulated by the facial muscle kinematic model include one or more of the following: orbicularis oculi, orbicularis oris, zygomaticus major, masseter, temporalis, depressor anguli oris, and platysma.

7. An electronic device, characterized in that, It includes a processor and a memory storing a computer program, wherein when the processor executes the computer program, it implements the intelligent interactive fusion method for three-dimensional surface scan data modeling as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent interactive fusion method for three-dimensional surface scan data modeling as described in any one of claims 1-6.

Images