Face restoration method, scanning device, computing device and storage medium

Abstract

The invention provides a face restoration method, scanning equipment, computing equipment and a storage medium. The method comprises the following steps: acquiring a face model of a patient, wherein the face model comprises a defect area; determining a mirror image plane of the face model, wherein the mirror image plane defines that the face model comprises a first part of the defect area and a second part of the defect area; a reference area is selected from the second part of the face model in a user-defined mode; performing mirroring on the reference area based on the mirroring plane to generate a mirroring area; and attaching the mirror image area to the defect area to generate a facial repair model. According to the method, the repair efficiency and the repair precision of the defect area can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction, and in particular to a facial repair method, a scanning device, a computing device and a storage medium. BACKGROUND

[0002] In facial defect repair, using three-dimensional digital technology for surgical simulation is a common method, such as virtually reconstructing the defect area by positioning the facial tissue of the mirror image healthy side of the patient. It is necessary to determine the healthy area according to the mirror image positioning of the defect area. However, the mirror image healthy area may not be the real healthy area, for example, there are other defects or it is not suitable as a repair basis. In addition, the generated repair body is only a rigid mirror image copy of the healthy side, which cannot be personalized and fine-tuned, resulting in distorted and unnatural repair simulation results, and it is difficult to meet the high clinical requirements of considering function and aesthetics. SUMMARY

[0003] A first aspect of the embodiments of the present application provides a facial repair method, the method comprising: obtaining a facial model of a patient, the facial model comprising a defect area; determining a mirror plane of the facial model, the mirror plane defining a first part of the facial model comprising the defect area and a second part of the facial model not comprising the defect area; selecting a reference area from the second part of the facial model based on the mirror plane; generating a mirror image area based on the mirror plane for the reference area; and attaching the mirror image area to the defect area to generate a facial repair model.

[0004] In some embodiments of the present application, the reference area is a non-mirror image part of the defect area in the second part, or the reference area is a mirror image part of the defect area in the second part.

[0005] In some embodiments of the present application, before generating the facial repair model, further comprising: aligning the edge curvature of the attached mirror image area and the defect area, and / or deleting the overlapping part of the attached mirror image area and the surrounding area of the defect area from the attached mirror image area, and / or changing the vertex normal vector of the edge area of the attached mirror image area.

[0006] In some embodiments of the present application, generating the facial repair model comprises: performing mesh fusion between the edge area of the attached mirror image area and the facial model, wherein the mesh fusion comprises selectable multi-level mesh fusion, each level of mesh fusion being configured to present different smoothing degrees and / or adapt to different facial structures.

[0007] In some embodiments of the present application, aligning the edge curvatures of the attached mirror region and the defect region comprises: determining an edge matching degree based on a first mesh curvature of an edge region of the attached mirror region and a second mesh curvature of an edge region of the defect region; in a case where the edge matching degree does not satisfy a preset matching degree, offsetting a vertex of the edge region of the attached mirror region.

[0008] In some embodiments of the present application, the reference region is determined by selecting a region corresponding to the second part of the face model.

[0009] In some embodiments of the present application, the reference region is determined by selecting a reference point in the second part of the face model.

[0010] In some embodiments of the present application, further comprising: determining the reference point in response to a clicking operation on the face model; determining an adjacent vertex adjacent to the reference point from mesh vertices corresponding to the face model; calculating a curvature difference between a mesh curvature of a position of the reference point and a mesh curvature of a position of the adjacent vertex; determining an adjacent vertex corresponding to the curvature difference less than a preset difference value, and constructing the reference region based on a closed region corresponding to the determined adjacent vertex.

[0011] In some embodiments of the present application, the reference region is determined by selecting a healthy region template of a face part corresponding to the defect region, and determining a boundary of the healthy region template based on a face contour of the patient.

[0012] In some embodiments of the present application, further comprising: changing the reference region based on the defect region so that the reference region and the defect region are adapted, and the changing comprises at least one of cropping the reference region, scaling the reference region, and scaling the cropped reference region.

[0013] In some embodiments of the present application, the reference region and / or the mirror region has a certain transparency, and the reference region is configured to be hidden or displayed.

[0014] In some embodiments of the present application, an edge region of the attached mirror region has a highlight effect.

[0015] In some embodiments of the present application, further comprising: in response to one or more of moving, rotating, and deforming the mirror region, generating a changed mirror region; and generating a face repair model by attaching the changed mirror region to the defect region.

[0016] In some embodiments of the present application, the deforming is performed by changing a shape of a lattice generated outside the mirror region.

[0017] In some embodiments of the present application, further comprising: in response to the deforming of the mirror region, changing a shape of a lattice generated outside the mirror region, or changing a shape of a first lattice corresponding to the mirror region, or changing a shape of a second lattice of the face model adjacent to an edge lattice of the mirror region.

[0018] In some embodiments of the present application, the mirror plane is a median sagittal plane of the patient.

[0019] In some embodiments of the present application, the mirror plane is determined by a plurality of touch actions on a display region corresponding to the face model.

[0020] In some embodiments of the present application, the plurality of touch actions are used to determine a plurality of non-collinear points on the display region.

[0021] In some embodiments of the present application, the plurality of touch actions are used to adjust an axis position of one or more axes of the display region to a target axis position.

[0022] A second aspect of the embodiments of the present application further provides a scanning device, the scanning device comprising a processor and a memory, the processor being configured to implement the face repairing method when executing a computer program stored in the memory.

[0023] A third aspect of the embodiments of the present application further provides a computing device, the computing device comprising a processor and a memory, the processor being configured to implement the face repairing method when executing a computer program stored in the memory.

[0024] A fourth aspect of the embodiments of the present application further provides a non-transitory computer readable storage medium, the non-transitory computer readable storage medium storing a computer program, the computer program being configured to implement the face repairing method when executed by a processor.

[0025] In the face repairing method provided in the application, a face model of a patient is obtained, and the face condition of the user is accurately restored through the face model, so as to subsequently perform repairing based on the face condition. A mirror plane of the face model is determined to establish a data basis for accurately mapping a healthy area to a defect area. Based on the determined mirror plane, the face model can be defined as two parts, i.e., a first part including the defect area and a second part not including the defect area. Then, a reference area is selected from the second part based on the healthy area determined based on the mirror plane, and the reference area is mirrored to generate a mirror area, which is the area mapped to the defect area, so as to improve the generation efficiency and accuracy of the mirror area. The mirror area is attached to the defect area to generate a face repairing model, so as to quickly and effectively repair the face model and improve the accuracy and efficiency of face defect repairing. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A structural schematic diagram of an example scanning device is shown.

[0027] Figure 2 A schematic diagram of an example scanning process is shown.

[0028] Figure 3 A flowchart of a face repairing method provided in an embodiment of the application is shown.

[0029] Figure 4 A front view corresponding to a face model provided in an embodiment of the application is shown.

[0030] Figure 5 A front view corresponding to a face model provided in another embodiment of the application is shown.

[0031] Figure 6 A schematic diagram of a position of a reference area on a face model provided in an embodiment of the application is shown.

[0032] Figure 7 A schematic diagram of a reference area after cutting provided in an embodiment of the application is shown.

[0033] Figure 8 A schematic diagram of a mirror area attached to a defect area provided in an embodiment of the application is shown.

[0034] Figure 9 A flowchart of determination of a reference area provided in an embodiment of the application is shown.

[0035] Figure 10 A flowchart of updating of a mirror area provided in an embodiment of the application is shown.

[0036] Figure 11 A structural diagram of a computing device provided in an embodiment of the application is shown. DETAILED DESCRIPTION

[0037] For the convenience of understanding, the descriptions of some concepts related to the embodiments of the present application are exemplarily given for reference.

[0038] It should be noted that “at least one” in the present application means one or more, and “multiple” means two or more than two. “And / or” describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms “first”, “second”, “third”, “fourth” and the like (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0039] In the current clinical and scientific research practice of facial defect repair, three-dimensional digital technology can be used for surgical simulation, and the scheme depends on selecting the corresponding healthy tissue of the defect area based on the mirror plane, and then mirroring the facial tissue structure of the healthy side of the patient to the defect area, so as to reconstruct a complete facial morphology in a virtual environment. However, this technical path has limitations in actual application.

[0040] One of the steps of the scheme is to determine the mirror symmetry plane, and in the related art, this process relies on the identification and positioning of facial anatomical landmark points. Due to the natural asymmetry of the human face, whether it is soft tissue contour or bony structure, a completely ideal symmetry is rare. This inherent asymmetry leads to deviations in the identification of facial feature points and the determination of the mirror symmetry plane, resulting in deviations in the selection of healthy tissue in the first mirror in the related art, and the process of mirroring the healthy tissue to the defect area cannot solve the above-mentioned deviations, so that the selected healthy area deviates from the true healthy area.

[0041] Secondly, in terms of repair effect, the logic of the related art to generate a repair body is essentially to rigidly and symmetrically copy and flip the morphology of the healthy side. This processing method ignores the individualized differences of facial morphology in micro and macro, as well as the actual conditions and functional requirements of the surrounding tissue of the defect area. The result is often manifested as a stiff repair simulation morphology, which is not coordinated with the overall face and does not look natural. It is difficult to meet the clinical requirements of modern maxillofacial repair, which pursues both physiological function and natural beauty.

[0042] Therefore, the embodiment of the present application proposes a face repairing method, a scanning device, a computing device and a storage medium, which can divide the face model into a first part including the defect area and a second part not including the defect area based on the mirror plane, then select the reference area from the second part by self-definition, and generate the mirror area according to the reference area. The reference area is selected by self-definition instead of being selected by mirroring the defect area, which reduces the mirroring process and avoids selecting the wrong healthy area due to the mirror plane, and improves the determination efficiency, flexibility of the reference area and the adaptation degree of the mirror area and the defect area.

[0043] The face repairing method provided by the embodiment of the present application can be applied to a scanning device. First, the structure of the scanning device is described below.

[0044] Referring to Figure 1 As shown, a scanning device 10 for generating two-dimensional and three-dimensional data and models of a patient's face is described herein, which can be configured or adjusted as described herein to generate two-dimensional and three-dimensional images, monochrome or color three-dimensional models of surface features of a patient's face, and used for facial, orthopedic, dental treatment planning, such as maxillofacial repair, temporomandibular joint treatment, orthodontic and prosthetic treatment planning. The exemplary scanning device 10 can be held by an operator, such as a doctor, or placed by, for example, a tripod, a support platform, etc., in some examples, the operator can move around the patient's face to capture 2D and 3D image data of the patient's face by holding, for example, a detachable handle (not shown) of the scanning device 10, in other examples, the operator can directly hold the housing 20 of the scanning device 10 to move around the patient's face to capture 2D and 3D image data of the patient's face, or fix the scanning device 10 by a tripod or the like, and the patient keeps the face facing the scanning device 10 to capture 2D and 3D image data of the patient's face.

[0045] Continuing to refer to Figure 1 As shown, the scanning device 10 can include a housing 20 and an imaging module 30 located in the housing 20. The imaging modules 30 can be arranged side by side with each other, so that the field of view of each imaging module at least partially overlaps. The imaging module 30 can include a plurality of depth cameras to capture 2D data of the patient's face, one or more color cameras 32 to capture high-quality color photos of the patient's face, and one or more projectors 33 to project light to the patient's face.

[0046] In some examples, the imaging module 30 can include at least two depth cameras working together to generate a point cloud of the patient's face. The depth cameras can use structured light, stereo vision matching, time-of-flight or other 3D imaging techniques to generate a point cloud representing the 3D positions of the surface of the patient's face.

[0047] Time-of-flight 3D imaging is a technique for range imaging, which measures the distance of each point in an image between a sensor (or camera) and the object being photographed, thus generating a 3D map or depth map of the scene. Infrared light is often used for this purpose, as it provides good distance information regardless of the visible light conditions and is invisible to the human eye. Time-of-flight refers to the time it takes for light to travel from the projector 33 to the object and back to the sensor, e.g. an image sensor.

[0048] Time-of-flight 3D imaging can include modulated ToF methods, such as Pulsed Modulation and Continuous Wave Modulation time-of-flight. Pulsed modulation directly measures the time-of-flight, which can generally achieve longer distance measurements. The projector 33 emits light in very short pulses, with fast rise and fall times, and employs high optical power lasers or laser diodes, which can very accurately measure the departure time of the emitted light and the arrival time of the reflected light.

[0049] With continuous wave modulation, the phase difference between the transmitted and received signals can be measured. The signal shape can be varied, e.g. sinusoidal, square wave, etc. The phase between the received and transmitted signals can be estimated by calculating the cross-correlation, which is directly related to the distance based on the known modulation frequency (typically between 10 MHz and 100 MHz).

[0050] Structured light 3D imaging is a method of capturing 3D information about an object or scene by projecting a known pattern (typically lines or grids, sometimes more complex patterns) onto the object or scene, and then observing the distortion of the pattern from different viewing angles.

[0051] A light projector in the projector 33, such as a digital micromirror device, a liquid crystal display projector, or an organic electroluminescent display projector, projects a structured pattern, e.g. a series of lines or grids, onto the object. This is typically done using infrared light (often near-infrared light) or visible light, e.g. blue light. The structured light pattern and its structure are known, and one or more cameras are placed at known angles relative to the projector, capturing images of the pattern as it falls on the object. Since the shape of the object affects the distortion of the pattern, these images contain information about the shape of the object.

[0052] Next, the scanning device 10 or other suitable computing device calculates the disparity between the known original pattern and the captured distorted pattern, which depends on the 3D shape of the object. Using a method of triangulation, the distance from the camera to each point in the scene is calculated. This forms a depth map or 3D point cloud of the object, with each point having specific coordinates in 3D space.

[0053] The point cloud data is then processed using specialized software to create a digital 3D model of the object. This model can be a simple geometric representation or a detailed model that includes texture and color information.

[0054] In some embodiments, the projector 33 can include a single light source, for example, a light source that emits one of infrared light, white light, blue light, or other visible monochromatic light. In other embodiments, the projector 33 is configured to emit light having a wavelength between 405 nm and 1100 nm. In other embodiments, the projector 33 can include two or three identical light sources, for example, two or three light sources that emit infrared light. Alternatively, the projector 33 can include two or three different light sources, for example, a first light source that emits infrared light and a second light source that emits white light, or a first light source that emits infrared light, a second light source that emits white light, and a third light source that emits blue light. The two or three identical light sources can be part of the same projector 33 or can be implemented as separate units (e.g., in additional light projector units), and likewise, the two or three different light sources can be part of the same projector 33 or can be implemented as separate units (e.g., in additional projector units).

[0055] In some embodiments, the imaging module 30 can also include another projector (not shown), for example, a speckle pattern projector, a stripe pattern projector, or an image projector, among others.

[0056] By way of example, the scanning device 10 can include multiple infrared cameras 31a, 31b, and 31c that can capture 3D image data simultaneously, for example, within 60 milliseconds, 30 milliseconds, or 10 milliseconds of each other. Simultaneously captured data can be combined together into a model or set of 3D data generated by each camera. Since the patient’s facial data is captured at close range (typically no more than 100 cm) and the scanning device is moving very little or is fixed, the 3D data from the infrared cameras 31a, 31b, and 31c that simultaneously captured images can be simply combined based on the known geometric arrangement of the scanning device.

[0057] In some embodiments, a pair of stereo imaging devices, for example, stereo imagers, can be used to generate a 3D point cloud of the face. A stereo imager pair can include a pair of cameras in a known spatial relationship that can simultaneously image an object from different angles. For example, a stereo imager pair can include a first stereo imager pair 31a, 31b, and / or a second stereo imager pair 31a, 31c, and / or a third stereo imager pair 31b, 31c. The stereo images pairs captured by the stereo imager pairs can be used to generate a point cloud that represents the three-dimensional locations of the surface of the face within the field of view of each stereo image pair.

[0058] The plurality of infrared cameras 31a, 31b, and 31c can include infrared light sensitive image sensors optically coupled to lenses such that the depth imaging device has a suitable focal length and field of view to capture the entire face of the patient from a distance of between 25 centimeters to 100 centimeters.

[0059] The plurality of infrared cameras 31a, 31b, and 31c can be arranged in a straight line on the imaging module 30 or can be staggered and not in a straight line. The center point distance between infrared camera 31a and infrared camera 31b can be different than the center point distance between infrared camera 31b and infrared camera 31c, for example, infrared cameras 31b and 31c are closer together and further away from infrared camera 31a. In another example, the center point distance between the two groups of infrared cameras can be the same.

[0060] In one example, the scanning device 10 can also include a fourth imaging device or more imaging devices that can be arranged in a circle (e.g., an ellipse), square, rectangle, diamond, or a diamond shape, with each imaging device at a vertex of the shape or spaced apart along an edge. In some embodiments, five imaging devices can be used. The five imaging devices can be arranged in a circle (e.g., an ellipse), square, rectangle, diamond, or a diamond shape, with four imaging devices at a vertex of the shape or spaced apart along an edge and the fifth imaging device in the middle of the shape.

[0061] The scanning device 10 can also include a color imaging device, such as color camera 32, that can include, for example, a color CMOS sensor optically coupled to a lens such that the color imaging device has a suitable focal length and field of view to capture the entire face of the patient from a distance of between 25 centimeters to 100 centimeters.

[0062] A data connection (such as a serial communication connection) between the scanning device 10 and one or more computer processors (not shown) can enable the transfer of data collected by the plurality of infrared cameras 31a, 31b, 31c and the color camera 32 so that it can be processed to derive 3D measurements of the surface of the object / object being scanned. The one or more computer processors can be implemented in a computing device associated with the scanning device 10 or, alternatively, can be part of the scanning device 10 itself.

[0063] The scanning device 10 can include one or more light sources 34 that surround the plurality of depth cameras, the light sources 34 can be configured to emit light of the same or similar wavelength as the projector 33, for example, both emit infrared light, for enhancing the clarity of the depth images captured by the depth cameras.

[0064] The scanning device 10 can include an illumination band 35 surrounding the plurality of depth cameras, for example, the illumination band 35 can surround the imaging module 30 in a ring shape, and the illumination band 35 can be configured to illuminate the patient's face with at least 1.5 times the ambient illumination, for example, at least 800 Lux, within the distance at which the patient's face can be captured. In some embodiments, the illumination band 35 can illuminate the face with at least 1000 Lux, or at least 1500 Lux, or at least 2500 Lux, so that the scanning device can capture clear facial images. In some embodiments, the scanning device 10 can further include an ambient light sensor to sense the real-time ambient brightness and adjust the output of the illumination band 35. In some embodiments, the illumination band 35 can be composed of LEDs.

[0065] In some examples, the scanning device 10 can further include a scanning button 40, for example, a short press of the scanning button 40 can control the scanning device 10 to start the scanning process of the patient's face, a second short press of the scanning button 40 can control the scanning device 10 to pause the scanning process of the patient's face, and a long press (usually more than 2 seconds) of the scanning button 40 can control the scanning device 10 to confirm the completion of the scanning process and save the scanning data, for example, the point cloud or the three-dimensional model.

[0066] In some examples, the scanning device 10 can further include a connection line 50, for example, a USB 2.0 or USB 3.0 data line, to connect a relay device or a computing device, etc., to provide power and / or support data transmission for the scanning device 10.

[0067] In other examples, the scanning device 10 can further include a built-in power source, for example, a rechargeable power source, and does not need to include the connection line 50, so the scanning device 10 can further include a data transmission chip, for example, a Bluetooth chip, a Wi-Fi chip, or a cellular chip, etc.

[0068] Figure 2 An example illustration of the handheld scanning device 10 capturing image data of the patient's 60 face is shown, for example, the imaging module 30 of the scanning device 10 can capture image data from the front of the patient's 60 face at a suitable capture distance, and if necessary, the scanning device 10 can be moved to capture image data from the left and right sides of the patient's 60 face. In other examples, the scanning device 10 is fixed, and the patient 60 can adjust the pose and face to face the imaging module 30 of the scanning device 10 directly.

[0069] It can be understood that after capturing the facial image of the patient 60, the scanning device 10 or its associated computing device can execute a computer program to process the image data to create a digital 3D model of the patient 60, and then the computer program can also fit the facial texture data collected by the color camera 32 to the digital 3D model to generate a final digital 3D model containing texture and color information.

[0070] In some examples, the patient's face can include defects such as concave, cracks, scars, spots, etc. The facial repair method or process or example described herein can provide a feasible repair method for the patient or medical staff, for example, try to capture the facial image of the patient with defects using the example scanning device 10 described herein, generate a digital facial model, and then execute the facial repair method described herein in the program interface of the scanning device 10 or its associated computing device. Figure 3 is a flowchart of the facial repair method provided by the embodiments of the present application, which can be applied in the scanning device (for example Figure 1 ) scanning device 10), and can also be applied to a computing device. According to different needs, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0071] Step S301, obtaining a facial model of a patient.

[0072] In some embodiments of the present application, the patient refers to an individual who needs to receive professional diagnosis, intervention or service in the field of medical and health management. In the treatment of maxillofacial repair, the patient can be a user who needs to create an accurate repair model through facial scanning data due to congenital defects or trauma.

[0073] In some embodiments of the present application, the patient is scanned by a scanning device to obtain candidate data corresponding to the patient's face, and the scanning device can be a facial scanning device. Since there can be situations that affect the quality of the scanning data during the scanning process, for example, the lack of clarity of the image corresponding to the scanning data causes the details (such as fine scars) to be unable to be accurately presented; the missing of part of the facial data due to the occlusion (such as hair) or scanning dead angle; the missing or deformation of the facial data when the scanning device scans the face in a non-frontal pose; the exaggerated expression of the patient changes the facial muscle and geometric shape, affecting the extraction of facial features.

[0074] Therefore, after obtaining the candidate data, the quality of the candidate data can be detected, which can be one or more of symmetry detection, detail resolution detection, expression deviation detection, data integrity detection, etc.

[0075] In an example, the morphological consistency is evaluated by calculating the mirror symmetry of the facial key anatomical landmarks in the candidate data. When the average position deviation of the key points on the left and right sides of the face exceeds a preset deviation value, it is determined that the candidate data has deformation or missing, and the candidate data does not meet the preset quality requirement.

[0076] In an example, the key region spatial sampling rate of the image or point cloud data in the candidate data is checked. When the pixel density or point spacing of the candidate data does not reach the minimum accuracy threshold required by the reconstruction algorithm, it is determined that the resolution is insufficient, and the candidate data does not meet the preset quality requirement.

[0077] In an example, the candidate data is fitted with a standard neutral expression model to quantify expression deviation. When the fitting residual or the deformation amplitude of a specific region (such as the perioral region or the interbrow region) exceeds a preset range, it is determined that the non-neutral expression is too large, and the candidate data does not meet the preset quality requirement.

[0078] In an example, the data integrity of the key anatomical regions (such as the defect edge and the surgical path) used for clinical planning in the candidate data is detected. When there are data holes in these key anatomical regions that affect the continuity and availability of the model, it is determined that the candidate data is incomplete, and the candidate data does not meet the preset quality requirement.

[0079] If the candidate data does not meet the preset quality requirement, the patient can be scanned again to reacquire the candidate data. If the candidate data meets the preset quality requirement, the candidate data is used as the facial data.

[0080] The above is only an example. In actual applications, quality detection is not necessarily required for the candidate data, and the candidate data obtained by scanning the patient's face can be directly used as the facial data.

[0081] In some embodiments of the present application, the facial data refers to the raw or preliminary processed (such as the quality detection described above) data about the patient's face directly acquired by an imaging device (such as a 3D structured light scanner, a laser radar, a multi-view stereo vision system, a depth camera, etc.). The facial data is processed using a three-dimensional reconstruction algorithm to generate point cloud data; and a facial model corresponding to the patient's face is constructed based on the point cloud data. The facial model can be a continuous, textured, and digital curved surface model that can be used for accurate measurement and analysis. The construction method can include one or more of triangular meshing, curved surface fitting, and topology optimization algorithms.

[0082] The patient's face can have congenital defects or injuries, such as concave, cracks, scars, and spots. The defect region in the facial model is used to represent the congenital defect or injury, and the facial model can include at least one defect region.

[0083] The personalized facial features of the patient are completely and accurately digitized and restored through the three-dimensional facial model, which provides an objective data basis for the personalized design and accurate manufacturing of the subsequent restoration body.

[0084] In step S302, a mirror plane of the facial model is determined.

[0085] In some embodiments of the present application, the mirror plane can be the median sagittal plane of the patient, which is a virtual vertical plane for accurately dividing the human face into two symmetrical halves. Therefore, the facial model can be divided into a first part including the defect area and a second part not including the defect area according to the mirror plane. In the front view of the facial model, the median sagittal plane can be displayed as a line (which can be a dashed line or a solid line) for representing the visual indication of the intersection line between the median sagittal plane and the surface of the facial model.

[0086] In combination with Figure 4 , Figure 4 is the front view corresponding to the facial model. As Figure 4 the dashed line is the median sagittal plane, the facial model is divided into a first part and a second part based on the median sagittal plane, and the defect area is the area indicated by M as Figure 4 .

[0087] In some embodiments of the present application, the mirror plane can be determined by a plurality of touch actions on the display area corresponding to the facial model. The plurality of touch actions are used to determine a plurality of non-collinear points on the display area.

[0088] In combination with Figure 5 , Figure 5 is the front view corresponding to the facial model, and the area indicated by M is the defect area. When the plurality of click operations performed by the user on the display area are detected, the positions indicated by each click operation are obtained, for example, Figure 5 the positions Q1, Q2 and Q3 shown in the figure. Figure 5 The positions indicated by the plurality of click operations form a plurality of non-collinear points, and the mirror plane is fitted by the plurality of non-collinear points, such as the dashed line shown in

[0089] In some embodiments of the present application, the plurality of touch actions are further used to adjust the axis position of one or more axes of the display area to a target axis position. Specifically, an initial plane can be defined on the display area in advance, which is perpendicular to the front view of the face model. The initial coordinate axis (X / Y / Z) and the initial angle (a / b / g) corresponding to the initial plane are determined, and the user can adjust the position and angle of the initial coordinate axis by dragging, and the plane equation Ax+By+Cz+D=0 is displayed in real time during the adjustment. When the axis position of one or more axes is adjusted to the target axis position, the adjusted plane can divide the face model into a first part and a second part, and the adjusted plane can be used as a reference plane for the face model. Figure 4 Or Figure 5 as shown by the dashed line.

[0090] Step S303, defining a reference area from the second part of the face model.

[0091] In some embodiments of the present application, the second part of the face model is defined as not including the defect area, and then a healthy area without defects can be determined from the second part as the reference area. In order to avoid the determination limitation, any area in the second part can be selected as the reference area.

[0092] In some embodiments of the present application, the mirror image part corresponding to the defect area of the first part can be determined from the second part, and the mirror image part is used as the reference area. The non-mirror image part corresponding to the defect area of the first part can also be determined from the second part, and the non-mirror image part is used as the reference area. The non-mirror image part can be at least part of the area in the second part except the mirror image part.

[0093] In some embodiments of the present application, the reference area can also be determined by selecting an area in the second part of the face model. The area selection can include polygon selection and free curve selection.

[0094] The polygon selection is used to determine the reference area, including: detecting a click operation on the second part of the face model, and generating a vertex based on the position indicated by the click operation. The vertex is used as the starting point and the ending point of drawing a closed polygon, and the closed polygon is determined as the reference area. The size and shape of the closed polygon are not limited in the present application.

[0095] In some embodiments of the present application, the process of drawing the closed polygon can be displayed in real time, for example, by displaying the boundary line of the closed polygon in red. In addition, a plurality of function modules can also be displayed, for example, a undo module, a delete module and a fine-tuning module. The undo module can be used to undo at least the previous operation, for example, to undo the generated vertex; the delete module can be used to delete the generated closed polygon; and the fine-tuning module is used to adjust the boundary line or the vertex of the closed polygon.

[0096] The reference region is determined by a free curve selection mode, including that a user can perform a drag operation on a mouse, and when the drag operation is stopped, a track generated on the second part based on the drag operation is acquired, and the start point and the end point of the track are automatically closed to form a selected region. In addition, when the start point and the end point of the track are automatically closed, it is necessary to ensure that the curve connecting the start point and the end point of the track does not belong to the first part. The selected region is taken as the reference region.

[0097] The generation process of the track can be displayed in real time, for example, the track is displayed in red. In addition, a plurality of function modules can also be displayed, for example, a cancel module, a delete module and a fine adjustment module. The cancel module can be used to cancel at least the previous operation, for example, to cancel the track; the delete module can be used to delete the selected region; and the fine adjustment module is used to adjust the boundary line of the selected region or the start point and the end point of the track.

[0098] In some embodiments of the present application, the reference region can also be determined by selecting a healthy region template matched with the facial part corresponding to the defect region, and changing the boundary of the healthy region template based on the patient's facial contour. A plurality of healthy region templates are pre-stored in the template library, for example, a left cheek template, a right lower jaw template, a lower half of the nose template, etc.

[0099] In response to the template calling operation, the healthy region template matched with the facial part corresponding to the defect region is queried from the template library. Since the healthy region template in the template library is a parameterized, standardized three-dimensional facial basic model, in order to adapt to the patient's facial contour, the boundary of the called healthy region template can be changed, and the changed healthy region template is taken as the reference region. The present application does not limit the changing manner, which can be one or more of magnification, reduction, cropping, etc.

[0100] In some embodiments of the present application, the reference region can also be determined by a reference point selected on the second part of the facial model. Specifically, the adjacent vertexes are determined based on the reference point, and the reference region is determined according to the mesh curvature of the reference point and the mesh curvature of the adjacent vertexes. The specific determination process can be combined with the embodiment shown in Figure 9 .

[0101] After the reference region is determined, the reference region can be highlighted. The part of the facial model which is not the reference region is set in the first layer, and the color feature when the facial model is constructed is maintained; the reference region is set in the second layer, and the transparency corresponding to the second layer is set to a preset value (such as 50%). In combination with Figure 6 , the area indicated by point P1 is the reference region, and the area indicated by point M is the defect region. The transparency of the second layer where the area indicated by point P1 is located is set to a preset value, so that the current color (such as blue) of the reference region is displayed.Figure 6 The gray shown covers the original color on the facial model (such as...). Figure 6 (as shown in white).

[0102] In other embodiments of this application, the reference area can be set to other transparency and color, for example, it can be set to blue with 60% transparency, and this application does not limit this. Additionally, the reference area can be individually hidden or shown on the display interface.

[0103] Step S304: Mirror the reference area based on the mirror plane to generate a mirror region.

[0104] In some embodiments of this application, mirroring refers to the operation of generating a geometrically symmetrical copy based on a certain plane or axis. For example, the mirrored area can be a copy of a missing area. Figure 6 As shown, the shape and other features of the determined reference region differ to some extent from the defective region. Therefore, before mirroring the reference region, it can be modified so that the mirrored reference region can fit the defective region.

[0105] Specifically, the reference region is cropped based on the missing area to generate a cropped reference region. For example, the reference region can be globally cropped by dragging the vertices of its bounding box, such as cropping the area near the eyes. Additionally, if the reference region includes parts like ears or eyes, these parts can be cropped separately. Alternatively, the reference region can be cropped based on a mirrored outline of the missing area, combined with... Figure 7 As shown, the area indicated by point P2 can be a pair of Figure 6 The region indicated by point P1 is generated after cropping, and the region indicated by point P2 is generated by cropping the mirror outline of the defective region. In order to generate a facial repair model later, the cropped reference region can be mirrored to generate a mirror region.

[0106] On the facial model's display interface, touch-sensitive sliders can be set, such as a mirror area scaling slider with a scaling range of 0.8-1.2 times. Based on the actual size of the missing area, the overall proportion of the reference area can be adjusted; for example, the reference area can be shrunk by a factor of 0.9. A mirrored region is then generated from the scaled reference area. Additionally, the cropped reference area can be scaled, and then the scaled, cropped reference area can be mirrored to generate another mirrored region.

[0107] The mirrored area can have a certain degree of transparency and can be hidden or shown individually.

[0108] Step S305: Attach the mirrored area to the defective area to generate a facial repair model.

[0109] In some embodiments of the present application, after the mirror region is attached to the defect region, the mirror region is fine-tuned in combination with Figure 8 As shown, the region indicated by P3 is the mirror region attached to the defect region. It can be visually observed that the defect region M and the mirror region P3 do not achieve precise fitting. Then one or more fine-tuning methods can be used to fine-tune the mirror region. The present application does not limit the execution order of the fine-tuning methods.

[0110] The first fine-tuning method is to align the edge curvatures of the attached mirror region and the defect region. The edge curvatures can be the grid curvatures of the corresponding region edges of the mirror region and the defect region, such as the radius, concave-convex direction, etc. Based on the first grid curvature of the edge region of the attached mirror region and the second grid curvature of the edge region of the defect region, the edge matching degree is determined. In the case that the edge matching degree does not satisfy the preset matching degree, the vertex of the edge region of the attached mirror region is offset until the edge matching degree satisfies the preset matching degree. For example, the edge of the attached mirror region can be adjusted by local vertex offset (e.g., offset amount ≤ 0.5 mm) to achieve seamless fitting.

[0111] The second fine-tuning method is to delete the overlapping part of the attached mirror region and the surrounding region of the defect region from the attached mirror region. Since the face belongs to a curved surface and has the property of incomplete symmetry, there may be overlapping grids after attachment. The grid of the attached mirror region and the grid of the defect region are detected. If it is determined that the grid of the attached mirror region and the grid of the defect region exist overlapping grids, the overlapping grids are deleted.

[0112] The third fine-tuning method is to change the vertex normal vector of the edge region of the attached mirror region. By changing the vertex normal vector of the edge region of the mirror region, the mirror region can be adapted to the edge and size of the defect region. The change can be made by dragging.

[0113] After the mirror region is fine-tuned by one or more fine-tuning methods described above, the edge region of the attached mirror region has a highlight display effect. Then, the grid fusion between the edge region of the attached mirror region and the face model can be performed. In addition, in the case that the attached defect region does not need to be fine-tuned, the grid fusion between the edge region of the attached mirror region and the face model can be directly performed.

[0114] The Poisson fusion algorithm can be used for grid fusion to generate a face repair model. Specifically, the transition region (width 5-10 mm) between the mirror region and the face model is subjected to grid smoothing processing. By adjusting the vertex normal vector of the transition region, the surface gloss and concave-convex texture of the mirror region are made consistent with the face model, and the splicing feeling is eliminated.

[0115] The mesh fusion can further include selectable multi-level mesh fusion, each level of mesh fusion being configured to present different smoothing degrees and adapt to different facial structures. For example, by supporting turning on a "transition area highlight" (displayed as a yellow semi-transparency), a user can visually check the fusion range, while providing a "fusion strength adjustment" (weak / medium / strong, corresponding to different smoothing degrees) to adapt to different facial structures (such as "strong fusion" for large-area regions such as cheeks, and "weak fusion" for fine regions such as nose wings).

[0116] Through the above embodiments, the facial data of the patient is acquired to construct a facial model, and the facial model is used to accurately restore the facial condition of the user, so that subsequent repair is performed based on the facial condition. A mirror plane of the facial model is determined to establish a data basis for accurately mapping a healthy region to a defect region. Based on the determined mirror plane, the facial model can be defined as two parts, a first part including the defect region and a second part not including the defect region. Then, a reference region is selected from the second part based on the healthy region provided by the mirror plane, and the reference region is mirrored to generate a mirror region, which is the region mapped to the defect region, so as to improve the generation efficiency and accuracy of the mirror region. The mirror region is attached to the defect region to generate a facial repair model, so as to quickly and effectively repair the facial model and improve the accuracy and efficiency of facial defect repair.

[0117] Figure 9 FIG. 1 is a flowchart of a process for determining a reference region according to an embodiment of the present application. The reference region can be determined by selecting a reference point in the second part of the facial model, as shown in FIG. 1, including the following steps. Figure 9

[0118] In step S901, a reference point is determined in response to a click operation on the facial model.

[0119] In some embodiments of the present application, the user can perform a click operation at any position of the non-defect region in the facial model, and the position where the user performs the click operation on the facial model is detected. In response to the click operation, the position on the facial model where the click operation is performed is taken as the reference point.

[0120] In step S902, adjacent vertices adjacent to the reference point are determined from the grid vertices corresponding to the facial model.

[0121] In some embodiments of the present application, the grid corresponding to the position of the reference point on the facial model is determined as a reference grid. Starting from the reference grid, adjacent grid vertices adjacent to the reference grid are determined from the grid vertices corresponding to the facial model, and the adjacent grid vertices are taken as the adjacent vertices.

[0122] ​By determining the adjacent vertex adjacent to the reference point, the adjacent vertex corresponding to the adjacent mesh in a proper range can be adaptively extracted according to the reference point, and the operation difficulty is reduced.

[0123] In step S903, a curvature difference between the mesh curvature of the position of the reference point and the mesh curvature of the position of the adjacent vertex is calculated.

[0124] In some embodiments of the present application, in order to ensure whether the determined adjacent vertex meets the requirements, a curvature difference between the mesh curvature of the position of the reference point and the mesh curvature of the position of the adjacent vertex can be calculated.

[0125] Specifically, if the position of the reference point is the position of the mesh vertex corresponding to the mesh where the reference point is located, the mesh curvature of the mesh vertex corresponding to the mesh where the reference point is located can be directly obtained, which is denoted as a first reference curvature; the mesh curvature of the position of the adjacent vertex is obtained, which is denoted as an adjacent curvature. The absolute difference between the first reference curvature and the adjacent curvature is calculated in a difference measurement manner, and the curvature difference is obtained.

[0126] In other embodiments of the present application, if the position of the reference point is the position of the non-mesh vertex (such as a patch) corresponding to the mesh where the reference point is located, the mesh curvature corresponding to the position of the reference point can be calculated according to the barycentric coordinates of the patch corresponding to the position of the reference point using an interpolation algorithm, which is denoted as a second reference curvature. The mesh curvature of the position of the adjacent vertex is obtained, which is denoted as an adjacent curvature. The absolute difference between the second reference curvature and the adjacent curvature is calculated, and the curvature difference is obtained.

[0127] In step S904, the adjacent vertex corresponding to the curvature difference smaller than the preset difference value is determined, and a reference region is constructed based on the closed region corresponding to the determined adjacent vertex.

[0128] In some embodiments of the present application, the adjacent vertex meeting the curvature requirement can be judged after the curvature difference is calculated each time. If the curvature difference corresponding to any adjacent vertex is smaller than the preset difference value, it indicates that the any adjacent vertex meets the curvature requirement, and if the curvature difference corresponding to any adjacent vertex is greater than or equal to the preset difference value, it indicates that the any adjacent vertex does not meet the curvature requirement. After the adjacent vertices corresponding to all curvature differences smaller than the preset difference value are determined, the closed region is constructed according to the adjacent vertices corresponding to all curvature differences smaller than the preset difference value, and the closed region is taken as the reference region. The preset difference value belongs to the range of 0.8 to 0.95, which can be dynamically adjusted according to actual application, and the present application does not limit this.

[0129] Through the above embodiments, the generation efficiency of the reference region can be improved, and the curvature change inside the reference region is ensured to be gentle by calculating the curvature difference, so as to avoid abrupt curvature jumps.

[0130] Figure 10is a flowchart of an updating process of a mirror region provided by an embodiment of the present application. As shown in the embodiment of Figure 10 The mirror region is adjusted to meet the contour difference requirements of different patients. The steps include the following.

[0131] In step S1001, a changed mirror region is generated in response to one or more of movement, rotation, and deformation of the mirror region.

[0132] In some embodiments of the present application, after the mirror region is determined, a three-axis drag arrow is displayed on the mirror region to facilitate movement adjustment. For example, the three axes include an X-axis, a Y-axis, and a Z-axis, the X-axis supports left-right direction movement, the Y-axis supports up-down direction movement, and the Z-axis supports front-back direction movement. The user can perform a drag operation on one or more of the three axes. In addition, a parameter input box for adjusting the three axes can also be set, for example, a specific movement distance and direction (such as X-axis left direction movement 0.1mm) are input in the parameter input box, and then the mirror region can be moved according to the input movement distance and direction.

[0133] In addition, the three-axis also supports rotation function, for example, a rotatable three-axis rotation ring is set to realize the rotation function. For example, rotating around the X-axis can realize the pitch rotation, rotating around the Y-axis can realize the left-right rotation, and rotating around the Z-axis can realize the twist rotation. The rotation function also supports adjusting the rotation center point, for example, the rotation center point is switched from the geometric center to a specified point on the defect edge, which can be the nasal ala vertex, and then the rotation around the nasal ala vertex can be realized to adapt to local fine adjustment.

[0134] In some embodiments of the present application, the deformation is performed by changing the shape of the lattice generated outside the mirror region. Specifically, the function of lattice deformation can be set for the mirror region. The function of lattice deformation can include changing the shape of the lattice generated outside the mirror region, or changing the shape of the first lattice corresponding to the mirror region, or changing the shape of the second lattice of the face model adjacent to the edge lattice of the mirror region.

[0135] The shape of the lattice generated outside the mirror region is changed as follows: without changing the first lattice corresponding to the mirror region and the second lattice corresponding to the face model, the lattice can be filled in the boundary area between the mirror region and the face model by filling the lattice. By changing the shape or number of the filled lattice, the shape of the lattice generated outside the mirror region is changed, so that the mirror region can smoothly transition to the face model through the filled lattice.

[0136] The shape of the first lattice corresponding to the mirror region is changed, and the implementation process is as follows: a lattice control box (default 2*2*2 lattice, which can be increased to 4*4*4) is generated around the mirror region, which is used to control the shape of the mirror region. In an example, the lattice of the mirror region is taken as the first lattice, and by dragging the lattice control box, the first lattice can be changed, thereby changing the shape of the mirror region (such as stretching the middle of the cheek and shrinking the edge of the lower jaw). During the deformation process, the grid topology is kept unchanged to avoid tearing of the patch.

[0137] The shape of the second lattice of the face model adjacent to the edge lattice of the mirror region is changed, and the implementation process is as follows: in the case that the size of the mirror region is small, it is difficult to adjust the lattice of the mirror region, and then the shape of the second lattice of the face model can be adjusted to realize the smooth transition of the face model to the mirror region.

[0138] In some other embodiments of the present application, the mirror region can also be locally deformed. For example, a brush-type deformation tool is provided to realize local deformation, and a brush radius (such as 1mm-5mm) and a deformation strength (weak / medium / strong) are set. The user can control the brush to paint on the surface of the mirror region, that is, local convexity / depression adjustment can be realized, for example, for the lips after mirroring, the thickness is made consistent with the opposite side by painting.

[0139] After the mirror region is changed by one or more of the above-mentioned ways, the mirror region before the change and the mirror region after the change can be previewed. By comparing the differences between the mirror region before the change and the mirror region after the change, it is supported to select and undo one or more steps in generating the mirror region after the change.

[0140] In step S1002, the changed mirror region is attached to the defect region to generate a face repair model.

[0141] In some embodiments of the present application, after the changed mirror region is attached to the defect region, fine tuning in step S305 as shown in Figure 3

[0142] In some other embodiments of the present application, after the changed mirror region is attached to the defect region, the preview of the attachment effect is supported, and multi-angle and multi-view preview operations are provided.

[0143] ​In an example, a view angle list corresponding to six standard view angles of "front", "left 45°", "right 45°", "side (left / right)", "top", and "bottom" is provided, and when it is detected that a user clicks a view angle switching operation, the view angle of the preview is switched. In addition, the user can customize the view angle, and the existing view angle list can be updated by customizing the view angle. For example, when it is detected that the user adjusts to an arbitrary view angle by dragging and clicks to save, the arbitrary view angle can be added to the view angle list to form a new view angle list.

[0144] In addition, partial enlargement (e.g., a certain region of the model is framed and automatically enlarged to full screen), partial reduction (e.g., a certain region of the model is framed and automatically reduced), and section viewing (e.g., a custom section plane is defined to display the internal grid structure of the model to verify the internal fit of the changed mirror region and the face model) and other functions are also supported.

[0145] Through the above embodiments, the personalized needs and fine adjustments of the user can be met, and the preview mode can be used to detect details and improve the smoothness of the face repair model.

[0146] In other embodiments of the present application, after the face repair model is generated, the compatibility of the first grid and the second grid corresponding to the defect region of the face repair model and the face model can be detected, for example, whether the grid patches are triangular patches, and whether the vertices are repeated. If there are problems such as non-triangular patches, it is determined that the compatibility is not met, and the grid formats of the first grid and the second grid can be unified (e.g., patch size deviation ≤0.1 mm) until the compatibility is met.

[0147] In the case where it is determined that the compatibility is met, the closed grid corresponding to the defect region can be determined according to the face repair model and the face model, and the repair map of the defect region corresponding to the face model can be constructed based on the closed grid.

[0148] Specifically, the closed grid can be generated by Boolean operation, specifically, difference operation logic is used, all the grids of the face repair model are taken as target grids, and all the grids of the face model are taken as tool grids. The grid difference between the target grid and the tool grid is calculated to obtain a closed grid containing only the defect region, and the closed grid of the defect region has no opening and no overlapping patch. Whether the closed grid of the defect region meets the closedness is detected, for example, if the closed grid of the defect region has an opening, it means that the closed grid of the defect region does not meet the closedness. If the closed grid of the defect region has no opening, it means that the closed grid of the defect region meets the closedness.

[0149] In a case where the closed mesh of the defect area meets the closed property, a standard format (such as STL, OBJ) of the closed mesh is exported to construct the restoration body. In a case where the closed mesh of the defect area does not meet the closed property, a face restoration model can be regenerated, or the closed mesh is regenerated until the closed mesh of the defect area meets the closed property.

[0150] The face restoration method provided by the embodiments of the present application can also be applied to a computing device, such as the computing device 1100 shown in FIG. 11. Figure 11

[0151] Figure 11 FIG. 11 is a structural diagram of a computing device provided by the embodiments of the present application. As shown in FIG. 11, the computing device 1100 can include a display device 1101, a communication module 1102, a memory 1103, a processor 1104, an input / output (I / O) interface 1105, and a bus 1106. The processor 1104 is coupled to the display device 1101, the communication module 1102, the memory 1103, and the I / O interface 1105 through the bus 1106. Figure 11

[0152] The display device 1101 can be a touch screen, which is a kind of inductive touchable liquid crystal display device. Alternatively, the display device 1101 can also be a non-touch screen. The display device 1101 can be used to display the processing result of the data sent by the scanning device 10.

[0153] The communication module 1102 can include a wired communication module and / or a wireless communication module. The wired communication module can provide one or more of the following wired communication solutions: Universal Serial Bus (USB), Controller Area Network (CAN) bus, and the like. The wireless communication module can provide one or more of the following wireless communication solutions: Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR) technology, and the like. ​​The memory 1103 can include one or more Random Access Memories (RAMs) and one or more Non-Volatile Memories (NVMs). The Random Access Memory can be directly readable and writable by the processor 1104, and can be used to store executable programs (e.g., machine instructions) of an operating system or other programs that are running, and can also be used to store data of users and applications, etc.

[0154] The Random Access Memory can include Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc.

[0155] The Non-Volatile Memory can also store executable programs and store data of users and applications, etc., and can be loaded in advance into the Random Access Memory for direct reading and writing by the processor 1104. The Non-Volatile Memory can include a magnetic disk storage device, a Flash Memory.

[0156] The memory 1103 is configured to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 1104. The one or more computer programs include a plurality of instructions, which when executed by the processor 1104, can implement the data desensitization operation performed on the computing device 1100.

[0157] In other embodiments, the computing device 1100 further includes an external memory interface for connecting an external memory, so as to expand the storage capacity of the computing device 1100.

[0158] The processor 1104 can include one or more processing units, for example: the processor 1104 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0159] The processor 1104 provides computing and control capabilities, for example, the processor 1104 is used to execute the computer program stored in the memory 1103 to realize the processing of the data sent by the scanning device 10.

[0160] The I / O interface 1105 is used to provide a channel for user input or output, for example, the I / O interface 1105 can be used to connect various input and output devices, for example, a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information, or make the information visualized.

[0161] The bus 1106 is used to provide at least a communication channel between the communication module 1102, the memory 1103, the processor 1104, and the I / O interface 1105 in the computing device 1100.

[0162] It can be understood that the structure of the embodiment of the present application does not constitute a specific limitation on the computing device 1100. In other embodiments of the present application, the computing device 1100 can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0163] The embodiment of the present application also provides a non-transitory computer readable storage medium, the non-transitory computer readable storage medium stores a computer program, the computer program includes program instructions, and the method implemented when the program instructions are executed can refer to the method in each of the above embodiments of the present application.

[0164] The computer readable storage medium can be an internal storage of the electronic device, such as a hard disk or a memory of the electronic device. The computer readable storage medium can also be an external storage of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like.

[0165] In some embodiments, the computer readable storage medium can include a program storage area and a data storage area. The program storage area can store an operating system, an application required by at least one function, and the like. The data storage area can store data created according to use of the electronic device, and the like.

[0166] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0167] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0168] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the apparatus / terminal device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0169] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0170] The above examples are only used to illustrate the technical solutions of the present application, but not limit the same; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A facial repair method, comprising: Obtain a facial model of the patient, the facial model including the defect area; Determine a mirror plane for the facial model, the mirror plane defining a first portion of the facial model including the defective region and a second portion excluding the defective region; Custom selection of reference areas from the second part of the facial model; The reference region is mirrored based on the mirror plane to generate a mirror region; The mirrored region is attached to the defective area to generate a facial repair model.

2. The method according to claim 1, wherein, The reference region is either a non-mirror portion of the defective region in the second part, or a mirror portion of the defective region in the second part.

3. The method according to claim 1, wherein, Before generating the facial reconstruction model, the following steps are also included: Align the edge curvature of the attached mirror region and the defective region, and / or, remove the overlapping portion of the attached mirror region and the surrounding region of the defective region from the attached mirror region, and / or, change the vertex normal vector of the edge region of the attached mirror region.

4. The method according to any one of claims 1 to 3, wherein, Generate a facial reconstruction model, including: Mesh fusion is performed between the edge region of the attached mirrored region and the facial model, wherein the mesh fusion includes selectable multi-level mesh fusion, each level of mesh fusion being configured to present different degrees of smoothness and / or adapt to different facial structures.

5. The method according to claim 3, wherein, Aligning the edge curvature of the attached mirror region and the defective region includes: The edge matching degree is determined based on the first grid curvature of the edge region of the attached mirror region and the second grid curvature of the edge region of the defective region; If the edge matching degree does not meet the preset matching degree, the vertex of the edge region of the attached mirror region is offset.

6. The method according to claim 1, wherein, The reference region is determined by selecting the region corresponding to the second part of the facial model.

7. The method according to claim 1, wherein, The reference area is determined by a reference point selected in the second part of the facial model.

8. The method according to claim 7, further comprising: In response to a click operation on the facial model, the reference point is determined; Determine the adjacent vertices of the reference point from the mesh vertices corresponding to the facial model; Calculate the curvature difference between the grid curvature at the position of the reference point and the grid curvature at the positions of the adjacent vertices; The adjacent vertices corresponding to curvature differences less than a preset difference are determined, and the reference region is constructed based on the closed regions corresponding to the determined adjacent vertices.

9. The method according to claim 1, wherein, The reference area is determined by selecting a healthy area template of the facial region corresponding to the defect area, and changing the boundary of the healthy area template based on the patient's facial contour.

10. The method according to claim 1, further comprising: The reference region is modified based on the defective region to adapt the reference region to the defective region. The modification includes at least one of cropping the reference region, scaling the reference region, and scaling the cropped reference region.

11. The method according to claim 1, wherein, The reference area and / or the mirrored area have a certain degree of transparency, and the reference area is configured to be hidden or displayed.

12. The method according to claim 1, wherein, The edge regions of the attached mirrored region have a highlighting effect.

13. The method according to claim 1, further comprising: In response to one or more of the movement, rotation, and deformation of the mirror region, a modified mirror region is generated; Generate a facial reconstruction model, including: The modified mirrored region is attached to the defective area to generate a facial repair model.

14. The method according to claim 13, wherein, The deformation is achieved by changing the shape of the lattice generated outside the mirror region.

15. The method of claim 14, further comprising: In response to deformation of the mirror region, the shape of the lattice generated outside the mirror region is changed, or the shape of the first lattice corresponding to the mirror region is changed, or the shape of the second lattice of the face model adjacent to the edge lattice of the mirror region is changed.

16. The method according to claim 1, wherein, The mirror plane is the midsagittal plane of the patient.

17. The method according to claim 1, wherein, The mirror plane is determined by multiple touch actions in the display area corresponding to the facial model.

18. The method according to claim 17, wherein, The multiple touch actions are used on the display area to determine multiple non-collinear points.

19. The method of claim 17, wherein, The multiple touch actions are used to adjust the position of one or more axes of the display area to the target axis position.

20. A scanning device comprising a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the facial repair method as described in any one of claims 1 to 19.

21. A computing device comprising a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the facial repair method as described in any one of claims 1 to 19.

22. A non-transitory computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the facial repair method as described in any one of claims 1 to 19.

Images