Image processing method, device, electronic device and storage medium

Abstract

The present invention provides an image processing method, device, electronic device and storage medium, the method comprising: obtaining relative position relationship information between an intraoral model and a neutral facial model of a target user; using base data of basic expressions to form a plurality of facial data corresponding to the expression state; using the facial data to update the expression state of the neutral facial model; based on the relative position relationship information, superimposing the part of the intraoral model that has not generated an aesthetic design and a pre-built tooth model onto the neutral facial model with the updated expression state, so as to obtain an expression image showing the aesthetic design effect. This solution dynamically shows the aesthetic restoration effect under different expressions to the target user by showing the expression image of the aesthetic design effect, thereby improving the display effect of the aesthetic restoration effect and the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art

[0002] As people pursue health and aesthetics, many patients with poor tooth shape hope to beautify their tooth shape through aesthetic restoration. Before formally performing aesthetic restoration on the teeth, the aesthetic restoration effect needs to be shown to the patients in advance, and the aesthetic restoration will only be formally performed on the patients' teeth after they are satisfied.

[0003] The current method of showing the aesthetic restoration effect to patients is to design a tooth model using aesthetic design software, and then superimpose the tooth model on the patient's facial image to simulate the patient's appearance after the aesthetic restoration. However, the aforementioned method can only display the designed tooth model statically, which makes it impossible for patients to fully understand the aesthetic restoration effect, resulting in poor display of the aesthetic restoration effect and poor patient experience. Summary of the invention

[0004] In view of this, embodiments of the present invention provide an image processing method, device, electronic device and storage medium to solve the problems of poor display effect of aesthetic restoration effect and poor patient experience in the current way of displaying aesthetic restoration effect to patients.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses an image processing method, the method comprising:

[0007] Acquire relative position relationship information between the intraoral model and the neutral facial model of the target user, wherein the neutral facial model is a three-dimensional facial model obtained by scanning the target user in a state of no expression;

[0008] Using basic data of basic expressions to form a plurality of facial data corresponding to the expression states, the basic data of basic expressions being constructed based on the neutral facial model;

[0009] Using the facial data, updating the expression state of the neutral facial model multiple times;

[0010] Based on the relative position relationship information, the part of the intraoral model for which no aesthetic design is generated and the pre-constructed tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image showing the aesthetic design effect.

[0011] Preferably, the intraoral model includes a maxillary model, a mandibular model and occlusal relationship data;

[0012] Based on the relative position relationship information, the part of the intraoral model for which no aesthetic design is generated and the pre-built tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image showing the aesthetic design effect, including:

[0013] Identify feature point information of the neutral facial model, calculate position changes of specific feature points among the feature points, and obtain a transformation matrix;

[0014] Based on the relative position relationship information and the transformation matrix, obtaining an updated position of the mandibular model in the neutral facial model with an updated expression state;

[0015] Based on the relative position relationship information and the updated position, the part of the intraoral model for which no aesthetic design is generated and the pre-constructed tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image showing the aesthetic design effect.

[0016] Preferably, the base data of the basic expression is used to form a plurality of facial data corresponding to the expression state, including:

[0017] Photographing the expression changes of the target user to obtain multiple frames of images to be processed;

[0018] A plurality of facial data corresponding to the plurality of frames of images to be processed are formed by using the plurality of frames of images to be processed and the basic data of basic expressions.

[0019] Preferably, using the base data of the multiple frames of images to be processed and basic expressions to form a plurality of facial data corresponding to the multiple frames of images to be processed includes:

[0020] Determining basic expressions related to the multiple frames of images to be processed and determining weight parameters of the basic expressions related to the multiple frames of images to be processed according to the multiple frames of images to be processed, wherein the multiple frames of images to be processed include RGB data and depth data;

[0021] A plurality of facial data corresponding to the plurality of frames of images to be processed are formed by using base data and weight parameters of basic expressions related to the plurality of frames of images to be processed.

[0022] Preferably, the portion of the intraoral model for which no aesthetic design is generated is the intraoral model outside the maxillary anterior teeth area, and the pre-constructed tooth model is the maxillary anterior teeth area tooth model.

[0023] Preferably, before superimposing the part of the intraoral model for which no aesthetic design is generated and the pre-built tooth model onto the neutral facial model with updated expression state, the method further comprises:

[0024] A tooth model designed for the target user is generated by means of aesthetic design software as the pre-built tooth model.

[0025] Preferably, the base data of the basic expression is used to form a plurality of facial data corresponding to the expression state, including:

[0026] A plurality of facial data corresponding to the expression states are formed by using weight parameters of basic expressions related to expression change images of other users, wherein the expression change images of the other users are obtained by photographing the expression changes of the other users.

[0027] Preferably, it also includes:

[0028] A three-dimensional animation is generated, wherein the three-dimensional animation includes: an animation with an aesthetic design effect formed by a plurality of frames of the expression images, and / or an animation without an aesthetic design effect formed by a plurality of frames of the images to be processed.

[0029] Preferably, obtaining the relative position relationship information between the target user's intra-oral model and the neutral face model includes:

[0030] Acquire a smiling face model, an intraoral model, and a neutral face model of a target user, wherein the smiling face model is a three-dimensional face model obtained by scanning the target user in a toothy smiling expression state;

[0031] The intra-oral model is aligned to the smiling face model, and the neutral face model is aligned to the smiling face model to obtain relative position relationship information between the intra-oral model and the neutral face model.

[0032] Preferably, the process of constructing basic data of basic expressions based on the neutral facial model includes:

[0033] Processing the mesh structure at the lips of the neutral facial model to separate the upper lip and the lower lip of the neutral facial model;

[0034] identifying feature point information of the neutral facial model after processing;

[0035] Based on the processed neutral facial model and the feature point information, base data of a plurality of basic expressions are constructed through deformation migration.

[0036] A second aspect of an embodiment of the present invention discloses an image processing device, the device comprising:

[0037] An acquisition unit, used to acquire relative position relationship information between an intra-oral model of a target user and a neutral face model, wherein the neutral face model is a three-dimensional face model obtained by scanning the target user in a state of no expression;

[0038] A forming unit, using base data of basic expressions to form a plurality of facial data corresponding to the expression states, wherein the base data of basic expressions is constructed based on the neutral facial model;

[0039] An updating unit, configured to update the expression state of the neutral facial model multiple times using the facial data;

[0040] A superimposition unit is used to superimpose the part of the intraoral model for which no aesthetic design is generated and the pre-constructed tooth model on the neutral facial model with updated expression state based on the relative position relationship information, so as to obtain an expression image showing the aesthetic design effect.

[0041] A third aspect of an embodiment of the present invention discloses an electronic device, comprising: a processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; and the memory is used to store a program, wherein the program is used to implement the image processing method disclosed in the first aspect of the embodiment of the present invention.

[0042] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the image processing method disclosed in the first aspect of the embodiment of the present invention.

[0043] Based on the above-mentioned embodiment of the present invention, an image processing method, device, electronic device and storage medium are provided. The method is: obtaining the relative position relationship information between the intraoral model and the neutral facial model of the target user; using the basic data of the basic expression to form a plurality of facial data corresponding to the expression state; using the facial data to update the expression state of the neutral facial model; based on the relative position relationship information, superimposing the part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model on the neutral facial model with the updated expression state to obtain an expression image showing the aesthetic design effect. In this scheme, the expression state of the neutral facial model is updated. The part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with the updated expression state, thereby obtaining an expression image showing the aesthetic design effect, thereby improving the display effect and user experience of the aesthetic restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;

[0046] Figure 2 An example diagram of an intraoral model provided by an embodiment of the present invention;

[0047] Figure 3 An example diagram of a smiling face model provided by an embodiment of the present invention;

[0048] Figure 4 An example diagram of a neutral face model provided by an embodiment of the present invention;

[0049] Figure 5 Another example diagram of a smiling face model provided by an embodiment of the present invention;

[0050] Figure 6 An example diagram of separating the upper lip and the lower lip of a neutral facial model provided by an embodiment of the present invention;

[0051] Figure 7 An example diagram of the lips of a neutral facial model before processing provided by an embodiment of the present invention;

[0052] Figure 8 An example diagram of the lips of a neutral facial model after processing provided by an embodiment of the present invention;

[0053] Fig. 9 An example diagram of facial feature points of a processed neutral face model provided by an embodiment of the present invention;

[0054] Fig.10 An example diagram of basic expressions provided by an embodiment of the present invention;

[0055] Fig.11 Another example diagram of a basic expression provided by an embodiment of the present invention;

[0056] Fig.12 An example diagram of a DSD aesthetic design model provided by an embodiment of the present invention;

[0057] Fig.13 An example diagram of a three-dimensional animated expression image provided by an embodiment of the present invention;

[0058] Fig.14 A structural block diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0061] As can be seen from the background technology, as people pursue health and aesthetics, many patients with bad tooth shapes hope to beautify their tooth shapes through aesthetic restoration, such as through digital smile design (DSD) to beautify their tooth shapes and show a brighter smile, thereby improving the patient's self-confidence. Before formally performing aesthetic restoration of teeth, it is necessary to show the aesthetic restoration effect to the patient in advance, and only after the patient is satisfied can the patient's teeth be formally restored aesthetically.

[0062] At present, the way to show the effect of aesthetic restoration to patients is to design a tooth model using aesthetic design software (such as DSD aesthetic design software), and then superimpose the tooth model on the patient's facial image to simulate the patient's appearance after the aesthetic restoration. In other words, at present, the patient's tooth design data (or tooth simulation design data) is simply superimposed on the patient's two-dimensional facial image to simulate the patient's appearance after the aesthetic restoration.

[0063] However, the existing method of displaying the aesthetic restoration effect can only display the designed tooth model statically, which results in that the patient cannot fully understand the aesthetic restoration effect, the display effect of the aesthetic restoration effect is poor and the patient experience is poor.

[0064] In order to solve the above problems, this solution proposes an image processing method, device, electronic device and storage medium to update the expression state of the neutral facial model. The part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with the updated expression state, so as to obtain an expression image showing the aesthetic design effect, thereby improving the display effect and user experience of the aesthetic restoration effect.

[0065] In other words, this solution brings a series of facial expressions and movements of patients who need aesthetic restoration into the aesthetic restoration effect display, and displays the aesthetic restoration effect of the patient under a series of facial expressions and movements in advance, thereby more vividly reflecting the effect picture after the patient completes the aesthetic restoration.

[0066] The present solution is described in detail below through various embodiments.

[0067] It should be noted that the subsequent explanation of this plan will involve scanning the target user's three-dimensional facial model and other personal information; before scanning the target user's three-dimensional facial model, this plan has informed the target user and obtained the target user's authorization to collect personal information, that is, this plan collects the target user's three-dimensional facial model and other personal information on the premise of complying with relevant laws and regulations.

[0068] See also Figure 1 , shows a flow chart of an image processing method provided by an embodiment of the present invention, the method comprising:

[0069] Step S101: Obtaining relative position relationship information between the intra-oral model and the neutral facial model of the target user.

[0070] It should be noted that the target users are patients who need aesthetic dental restoration.

[0071] In the process of implementing step S101, a smiling face model, an intraoral model and a neutral face model of the target user are obtained, wherein the smiling face model is a three-dimensional face model scanned when the target user is in a toothy smiling expression state, the neutral face model is a three-dimensional face model scanned when the target user is in an expressionless state, and the intraoral model includes a maxillary model, a mandibular model and occlusal relationship data.

[0072] The method of obtaining the smiling face model, intraoral model and neutral face model of the target user is: scanning the target user's mouth by a scanning device to obtain the target user's intraoral model; specifically, scanning the upper jaw and lower jaw in the target user's mouth by a scanning device to obtain the target user's upper jaw model and lower jaw model respectively, and scanning to obtain occlusal relationship data.

[0073] It should be noted that in the process of scanning the target user's occlusion data, the target user's upper and lower jaws need to be aligned. The scanned occlusion data can be used as a reference for the "change in the occlusion position of the upper and lower jaws" in the subsequent expression change processing.

[0074] For example: Scan to get Figure 2 The intraoral model of the target user is shown, and the intraoral model includes a maxillary model, a mandibular model and occlusal relationship data.

[0075] When the target user is in a toothy smiling expression state, the three-dimensional facial model of the target user is scanned by a scanning device to obtain the smiling facial model of the target user.

[0076] It should be noted that, during the process of scanning the target user's smiling facial model, the target user's teeth need to maintain a normal occlusion relationship (the upper jaw and the lower jaw need to be aligned with each other), that is, the occlusion relationship corresponds to the occlusion relationship when the intraoral model is obtained.

[0077] For example, when the target user's teeth maintain a normal occlusion relationship and the target user is in a toothy smile, the scanned image is as follows: Figure 3 The smiling face model of the target user is shown.

[0078] The target user is in a state of no expression (or the target user's mouth is naturally closed), and the three-dimensional facial model of the target user is scanned by a scanning device to obtain a neutral facial model of the target user.

[0079] For example, the target user is in a normal state with his mouth closed, and the scan results are as follows: Figure 4 The neutral facial model of the target user is shown.

[0080] The smiling face model, intraoral model and neutral face model of the target user can be scanned and acquired in the above manner.

[0081] It should be noted that the data collected by different scanning devices are usually not in an aligned state, and the data collected multiple times by the same scanning device are usually not in an aligned state; therefore, after scanning and obtaining the smiling face model, intraoral model and neutral face model of the target user, it is necessary to align the smiling face model, intraoral model and neutral face model of the target user.

[0082] In a specific implementation, the intra-oral model of the target user is aligned to the smiling face model of the target user, and the neutral face model of the target user is aligned to the smiling face model of the target user to obtain the relative position relationship information between the intra-oral model and the neutral face model.

[0083] That is to say, the target user's intra-oral model is first aligned to the smiling face model, and then the neutral face model is aligned to the smiling face model. This indirectly aligns the intra-oral model to the neutral face model, so that the relative position relationship information between the intra-oral model and the neutral face model can be obtained.

[0084] It should be noted that the alignment operations of "aligning the intraoral model to the smiling facial model" and "aligning the neutral facial model to the smiling facial model" can be implemented using a grid matching algorithm, that is, the intraoral model is aligned to the smiling facial model, and the neutral facial model is aligned to the smiling facial model through a grid matching algorithm.

[0085] In some embodiments, the grid matching algorithm includes a grid coarse matching algorithm, a grid fine matching algorithm, etc. For example, the grid matching algorithm is a grid matching algorithm such as PCA and ICP.

[0086] To better understand the alignment operation mentioned above, the following is an example: Figure 5 The tooth information on the smiling face model (toothy smile) of the target user is shown, and the maxillary model and mandibular model of the target user are aligned to the smiling face model; then, the neutral face model of the target user is aligned to the smiling face model; so far, the neutral face model, the maxillary model, and the mandibular model are indirectly aligned through the smiling face model of the target user, so that the relative position relationship information between the intra-oral model and the neutral face model can be obtained; the relative position relationship information can reflect the correct position relationship between the tooth model and the neutral face model.

[0087] Step S102: Using the base data of the basic expression, a plurality of facial data corresponding to the expression state are formed.

[0088] It should be noted that the basic data of the basic expression is constructed based on the neutral facial model; in some embodiments, the specific method of constructing the basic data of the basic expression based on the neutral facial model of the target user is:

[0089] Processing the mesh structure at the lips of the neutral facial model to separate the upper lip and the lower lip of the neutral facial model; identifying feature point information of the processed neutral facial model (the neutral facial model after the upper lip and the lower lip are separated); and constructing base data of multiple basic expressions through deformation migration based on the processed neutral facial model and the feature point information.

[0090] Specifically, the mesh structure at the lips of the neutral face model is processed by a mesh geometry processing algorithm, so as to separate the upper lip and the lower lip of the neutral face model;

[0091] For example, by processing the mesh structure of the lips of the neutral face model through the mesh geometry processing algorithm, we can get the following Figure 6 The example diagram of separating the upper lip and the lower lip of the neutral face model is shown; the situation at the lips of the neutral face model before processing is as follows Figure 7 As shown, the lips of the neutral face model after processing are as follows Figure 8 shown.

[0092] After the upper lip and the lower lip of the neutral face model are separated, facial feature points of the processed neutral face model are identified by AI recognition (or by designated software) to obtain corresponding feature point information;

[0093] For example: AI recognition can be used to identify Fig. 9 The facial feature points of the neutral facial model are shown, thereby obtaining the corresponding feature point information.

[0094] The processed neutral facial model and the feature point information of the neutral facial model obtained through recognition are used to construct base data of the basic expression of the target user through deformation migration, wherein the basic expression can also be called an expression base.

[0095] The base data of multiple basic expressions are constructed by deformation migration of expression base templates. The expression base template library contains at least single expressions such as left eye blink, right eye blink, open mouth, and upturned mouth. The expression base template is selected from the expression base template library, and deformation migration is used to construct the base data of multiple basic expressions corresponding to the target user. Fig.10 This is an example of the basic expression "open mouth". Fig.11 This is an example of the basic expression "tilted lips"; when using these basic expressions, select the corresponding basic expression from these basic expressions to use.

[0096] The constructed base data of the basic expressions of the target user are numbered, for example, A1, A2 and A3 are used to represent the base data of different basic expressions respectively; the constructed base data of each basic expression has the same grid topology as the neutral facial model.

[0097] In the specific implementation of step S102, the base data of the basic expressions pre-constructed above are used to form multiple facial data corresponding to the expression states; in the specific implementation, this solution provides at least two ways of forming facial data, which are respectively described below with examples.

[0098] The first method of forming facial data is: photographing the changes in facial expressions of the target user to obtain multiple frames of images to be processed; using the multiple frames of images to be processed and the basic data of basic expressions to form multiple facial data corresponding to the multiple frames of images to be processed, wherein the facial data corresponding to the images to be processed are composed of the basic data of the basic expressions and weight parameters.

[0099] Specifically, the target user's facial expression changes are photographed to obtain multiple frames of images to be processed (expression data) that can be used for expression tracking calculations. The images to be processed contain RGB data and depth data (Depth data), where RGB refers to red, green, and blue (red, green, and blue).

[0100] When the target user's expression changes, each frame of the image to be processed is collected for expression tracking calculation.

[0101] In a specific implementation, a method of forming multiple facial data using base data of multiple frames of images to be processed and basic expressions is as follows: based on the multiple frames of images to be processed, basic expressions related to the multiple frames of images to be processed are determined, and weight parameters of the basic expressions related to the multiple frames of images to be processed are determined, wherein the multiple frames of images to be processed include RGB data and depth data; and multiple facial data corresponding to the multiple frames of images to be processed are formed using the base data and weight parameters of the basic expressions related to the multiple frames of images to be processed.

[0102] It should be noted that multiple frames of images to be processed can be captured, and each frame of the images to be processed can form a corresponding piece of facial data, that is, one frame of the images to be processed can obtain one piece of facial data.

[0103] It should be further explained that the image to be processed contains RGB data and depth data (Depth data); for each frame of the image to be processed, the expression state of a frame of three-dimensional facial model can be obtained through the RGB data and Depth data in the frame of the image to be processed; the expression state of each frame of the three-dimensional facial model can be expressed by the base data and weight parameters of the basic expression related to the frame of the image to be processed, and the expression state expressed by the base data and weight parameters of the basic expression related to the frame of the image to be processed is the facial data mentioned above (denoted as A).

[0104] In summary, for each frame of image to be processed (each frame of RGB data and Depth data), facial data A corresponding to the frame of image to be processed is formed. The facial data A can be constructed by multiplying the base data of a finite number of basic expressions in the pre-constructed basic expressions by a certain weight parameter, for example, A=c1*A1+c5*A5+c8*A8, A1, A5 and A8 are the base data of the basic expressions related to the frame of image to be processed, c1, c5 and c8 are the weight parameters corresponding to A1, A5 and A8; the weight parameter ci corresponding to the base data of each basic expression is obtained, and the sum of ci is 1.

[0105] In summary, the coordinates of each point of the facial data A corresponding to each frame of the image to be processed are obtained by multiplying the coordinate value corresponding to Ai by ci and summing them up, where Ai is the base data of the basic expression i, and ci is the weight parameter corresponding to the base data of the basic expression i.

[0106] The facial data A formed in the above manner has the same mesh topology as the neutral facial model.

[0107] The above is the description of the first method of forming facial data.

[0108] The second way to form facial data: there is no need to capture the target user's expression changes (that is, there is no need to capture the image to be processed or to collect the target user's dynamic expression), and the expression change images of other users are used to form multiple facial data.

[0109] In a specific implementation, the base data and weight parameters of basic expressions related to the expression change images of other users are utilized to form multiple facial data corresponding to the expression states (used as the facial data of the target user). That is, there is no need to shoot the expression changes of the target user. The base data and weight parameters of basic expressions related to the expression change images of other users can be used to form multiple facial data of the target user. It should be noted that utilizing the base data of basic expressions related to the expression change images of other users means utilizing the base data of basic expressions related to the expression change images of other users to determine the expression base template corresponding to the weight parameters. Through the corresponding expression base template, when the target user utilizes the weight parameters of other users, the expression base template corresponding to each weight parameter and the corresponding constructed base data of the basic expression of the target user can be determined to form the facial data of the target user.

[0110] The base data of basic expressions related to the expression change images of other users are constructed based on a neutral face model, and the expression change images of other users are obtained by photographing the expression changes of other users.

[0111] In a practical application, after constructing the base data of each basic expression through the neutral facial model of the target user, the facial data of the target user can be formed using the weight parameters of preset values, or using the weight parameters obtained from the dynamic expressions of other users. The facial data of the target user can be formed without collecting the dynamic expressions of the target user.

[0112] For example: Assume that the target user is user B; obtain the weight parameter c obtained from the dynamic expression of user A, and then use the weight parameter c obtained from the dynamic expression of user A to form the facial data of user B. User B does not need to collect dynamic expressions.

[0113] In another practical application, the combination of basic data of basic expressions can be set by changing the weight parameters, thereby forming multiple facial data corresponding to the expression state (equivalent to using weight parameters and base data to set facial data). The aesthetic restoration effect of the target user under different facial expression changes can be dynamically simulated without collecting the dynamic expression of the target user.

[0114] It should be noted that when using expression images of other users to form the facial data of the target user, you can use only weight parameters, or you can use base data and weight parameters; however, when forming the facial data of the target user, the base data used is generated by the neutral facial model of the target user, and the base data of other users is only used as a reference expression for selecting the base data of the target user corresponding to the weight parameters, which can make the facial data of the target user formed more vivid and natural.

[0115] Step S103: using the facial data to update the expression state of the neutral facial model multiple times.

[0116] In the specific implementation of step S103, after the facial data corresponding to each frame of the image to be processed is formed, the facial data corresponding to each frame of the image to be processed is used to update the expression state of the neutral facial model multiple times.

[0117] Step S104: Based on the relative position relationship information, the part of the intraoral model where no aesthetic design is generated and the pre-built tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image showing the aesthetic design effect.

[0118] In some embodiments, a tooth model of a target user after aesthetic restoration (referred to as a pre-constructed tooth model) is pre-constructed. Specifically, a tooth model designed for the target user is generated as the pre-constructed tooth model through aesthetic design software;

[0119] For example: through DSD aesthetic design software, generate Fig.12 The DSD aesthetic design model of the maxillary anterior teeth area of ​​the target user is shown, and the DSD aesthetic design model is a pre-built tooth model.

[0120] It should be noted that the pre-built tooth model can be a texture mapping model, or other types of aesthetic design models, and the specific situation of the pre-built tooth model is not limited here. The texture of each frame of the expression image obtained subsequently comes from the neutral facial model, and the texture coordinates and the corresponding uv correspondence (texture coordinate correspondence) are the same as the neutral facial model of the target user. When the expression of the target user changes, the texture will also change, and then the calculation adjustment processing will be performed based on the texture source to obtain the corresponding texture after the expression change.

[0121] In the process of implementing step S104, based on the relative position relationship information between the intraoral model and the neutral facial model, the part of the intraoral model where the aesthetic design is not generated and the pre-built tooth model are superimposed on the neutral facial model with updated expression state in each frame to obtain a multi-frame expression image (three-dimensional image) showing the aesthetic design effect.

[0122] It should be noted that the expression state of the neutral facial model is updated through the facial data corresponding to each frame of the image to be processed, so each frame of the image to be processed can obtain a corresponding frame of expression image, which shows the neutral facial model with expression state and superimposed with the part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model.

[0123] It should be noted that the intraoral model includes the maxillary model, the mandibular model and the occlusal relationship data. During the change of facial expression, the mandible should dynamically change its position following the change of expression. The transformation matrix can be calculated by the position change of specific feature points of the face during the change of expression. The transformation matrix is ​​implemented as a matrix of mandibular tooth changes. The transformation matrix can calculate the position of the mandibular model in the neutral facial model with updated expression state.

[0124] In some embodiments, a specific method of obtaining an expression image showing an aesthetic design effect is as follows: identifying feature point information of a neutral facial model, calculating a position change of a specific feature point among the feature points, and obtaining a transformation matrix, wherein the specific feature point may be a feature point of a mandibular region of the face; based on the relative position relationship information and the transformation matrix, obtaining an updated position of the mandibular model in the neutral facial model with an updated expression state;

[0125] Based on the relative position relationship information and the updated position, the part of the intraoral model where no aesthetic design is generated and the pre-built tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image showing the aesthetic design effect.

[0126] In other embodiments, the portion of the intraoral model for which no aesthetic design is generated is the intraoral model outside the maxillary anterior teeth area, and the pre-constructed tooth model is the maxillary anterior teeth area tooth model.

[0127] In order to provide better display effects to target users, three-dimensional animations can be generated and played, thereby providing target users with more diverse display effects.

[0128] In some embodiments, a three-dimensional animation is generated, which includes: an animation with aesthetic design effects (animation with DSD) composed of multiple frames of expression images, and / or an animation without aesthetic design effects (animation without DSD) composed of multiple frames of images to be processed.

[0129] If the generated three-dimensional animation includes an animation with aesthetic design effects composed of multiple frames of expression images, then playing the three-dimensional animation can dynamically simulate the aesthetic restoration effects of the target user under different facial expression changes.

[0130] For example, play Fig.13Three frames of expression images in the three-dimensional animation shown dynamically simulate the aesthetic restoration effect of the target user under different facial expression changes.

[0131] In an animation with aesthetic design effects composed of multiple frames of expression images, from the expression image of the starting frame to the expression image of the ending frame, the facial data corresponding to each frame of the image to be processed is used in turn to update the expression state of the neutral facial model, and the part where the aesthetic design effect is not generated and the pre-built tooth model are superimposed, so as to achieve dynamic simulation of the aesthetic restoration effect of the target user under different facial expression changes; the target user can use this animation to view the aesthetic restoration effect under different facial expression changes from multiple angles, and simulate and show the overall effect after aesthetic design in advance.

[0132] In the embodiment of the present invention, the expression state of the neutral facial model is updated. The part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with the updated expression state, thereby obtaining an expression image constituting a three-dimensional animation. The three-dimensional animation is played to dynamically display the aesthetic restoration effects under different expressions to the target user, thereby improving the display effect of the aesthetic restoration effect and the user experience.

[0133] Corresponding to an image processing method provided by the above embodiment of the present invention, see Fig.14 , an embodiment of the present invention further provides a structural block diagram of an image processing device, the image processing device comprising: an acquisition unit 100, a formation unit 200, an update unit 300 and a superposition unit 400;

[0134] The acquisition unit 100 is used to acquire the relative position relationship information between the intra-oral model and the neutral face model of the target user. The neutral face model is a three-dimensional face model scanned when the target user is in a state of no expression.

[0135] The forming unit 200 is used to form a plurality of facial data corresponding to the expression state by using the basic data of the basic expression, wherein the basic data of the basic expression is constructed based on the neutral facial model.

[0136] The updating unit 300 is used to update the expression state of the neutral facial model multiple times using the facial data.

[0137] The superimposition unit 400 is used to superimpose the part of the intraoral model without aesthetic design and the pre-built tooth model on the neutral facial model with updated expression state based on the relative position relationship information to obtain an expression image showing the aesthetic design effect.

[0138] In some embodiments, the portion of the intraoral model for which no aesthetic design is generated is the intraoral model outside the maxillary anterior teeth area, and the pre-constructed tooth model is the maxillary anterior teeth area tooth model.

[0139] In the embodiment of the present invention, the expression state of the neutral facial model is updated. The part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with the updated expression state, thereby obtaining an expression image constituting a three-dimensional animation. The three-dimensional animation is played to dynamically display the aesthetic restoration effects under different expressions to the target user, thereby improving the display effect of the aesthetic restoration effect and the user experience.

[0140] Preferably, combined Fig.14 The content shown is that the intraoral model includes the maxillary model, the mandibular model and the occlusal relationship data, and the superposition unit 400 includes an identification subunit, a determination subunit and a superposition subunit. The execution principle of each subunit is as follows:

[0141] The recognition subunit is used to recognize the feature point information of the neutral face model, calculate the position change of specific feature points in the feature points, and obtain the transformation matrix.

[0142] A subunit is determined, which is used to obtain an updated position of the mandibular model in the neutral facial model with an updated expression state based on the relative position relationship information and the transformation matrix.

[0143] The superposition subunit is used to superimpose the part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model on the neutral facial model with updated expression state based on the relative position relationship information and the updated position, so as to obtain an expression image showing the aesthetic design effect.

[0144] Preferably, combined Fig.14 As shown, the forming unit 200 is specifically used to: use the weight parameters of the basic expressions related to the expression change images of other users to form multiple facial data corresponding to the expression states, wherein the expression change images of other users are obtained by photographing the expression changes of other users.

[0145] Preferably, combined Fig.14 The content shown, the forming unit 200 includes a shooting subunit and a forming subunit, and the execution principle of each subunit is as follows:

[0146] The shooting subunit is used to shoot the changes in the expression of the target user to obtain multiple frames of images to be processed.

[0147] The forming subunit is used to form a plurality of facial data corresponding to the plurality of frames of images to be processed by using the base data of the plurality of frames of images to be processed and the basic expressions.

[0148] In a specific implementation, the forming subunit is specifically used to: determine the basic expressions related to the multiple frames of images to be processed, and determine the weight parameters of the basic expressions related to the multiple frames of images to be processed, according to the multiple frames of images to be processed, wherein the multiple frames of images to be processed include RGB data and depth data. The base data and weight parameters of the basic expressions related to the multiple frames of images to be processed are used to form a plurality of facial data corresponding to the multiple frames of images to be processed.

[0149] Accordingly, the image processing device further includes:

[0150] The generating unit is used to generate a three-dimensional animation, wherein the three-dimensional animation includes: an animation with aesthetic design effects composed of multiple frames of expression images, and / or an animation without aesthetic design effects composed of multiple frames of images to be processed.

[0151] Preferably, combined Fig.14 The image processing device further comprises:

[0152] The design unit is used to generate a tooth model designed for a target user as a pre-built tooth model through aesthetic design software.

[0153] Preferably, combined Fig.14 As shown in the figure, the acquisition unit 100 includes an acquisition subunit and an alignment subunit, and the execution principle of each subunit is as follows:

[0154] The acquisition subunit is used to acquire the smiling face model, intraoral model and neutral face model of the target user. The smiling face model is a three-dimensional face model scanned when the target user is in a toothy smiling expression state.

[0155] The alignment subunit is used to align the intra-oral model to the smiling face model, and to align the neutral face model to the smiling face model, so as to obtain the relative position relationship information between the intra-oral model and the neutral face model.

[0156] Preferably, combined Fig.14 The content shown, the formation unit 200 includes a processing subunit, an identification subunit and a construction subunit, and the execution principle of each subunit is as follows:

[0157] A processing subunit is used for processing the mesh structure at the lips of the neutral face model to separate the upper lip and the lower lip of the neutral face model.

[0158] The recognition subunit is used to recognize feature point information of the processed neutral face model.

[0159] A sub-unit is constructed to construct base data of multiple basic expressions through deformation migration based on the processed neutral facial model and feature point information.

[0160] Preferably, an embodiment of the present invention further provides an electronic device, comprising: a processor and a memory, the processor and the memory being connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; the memory is used to store the program, and the program is used to implement the image processing method disclosed in the above method embodiment.

[0161] Preferably, an embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the image processing method disclosed in the above method embodiment.

[0162] In summary, the embodiments of the present invention provide an image processing method, device, electronic device and storage medium to update the expression state of a neutral facial model. The part of the intraoral model that has not generated the aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with the updated expression state, thereby obtaining an expression image showing the aesthetic design effect, thereby improving the display effect and user experience of the aesthetic restoration effect.

[0163] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0164] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0165] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: The method comprises: Acquiring relative position relationship information between an intraoral model and a neutral face model of a target user, including: acquiring a smiling face model, an intraoral model and a neutral face model of the target user, wherein the smiling face model is a three-dimensional face model scanned when the target user is in a toothy smiling expression state; aligning the intraoral model to the smiling face model, and aligning the neutral face model to the smiling face model, so as to obtain relative position relationship information between the intraoral model and the neutral face model, wherein the neutral face model is a three-dimensional face model scanned when the target user is in a non-expression state, and the intraoral model includes an upper jaw model, a lower jaw model and occlusion relationship data; Using basic data of basic expressions to form a plurality of facial data corresponding to the expression states, the basic data of basic expressions being constructed based on the neutral facial model; Using the facial data, updating the expression state of the neutral facial model multiple times; Based on the relative position relationship information, the part of the intraoral model for which no aesthetic design is generated and the pre-constructed tooth model are superimposed on the neutral facial model with an updated expression state to obtain an expression image showing the aesthetic design effect, which includes: identifying feature point information of the neutral facial model, calculating the position change of specific feature points among the feature points, and obtaining a transformation matrix; based on the relative position relationship information and the transformation matrix, obtaining the updated position of the mandibular model in the neutral facial model with an updated expression state; based on the relative position relationship information and the updated position, the part of the intraoral model for which no aesthetic design is generated and the pre-constructed tooth model are superimposed on the neutral facial model with an updated expression state to obtain an expression image showing the aesthetic design effect.

2. The method according to claim 1, characterized in that Using the basic data of basic expressions, multiple facial data corresponding to the expression states are formed, including: Photographing the expression changes of the target user to obtain multiple frames of images to be processed; A plurality of facial data corresponding to the plurality of frames of images to be processed are formed by using the plurality of frames of images to be processed and the basic data of basic expressions.

3. The method according to claim 2, characterized in that Using the multiple frames of images to be processed and the base data of the basic expressions, a plurality of facial data corresponding to the multiple frames of images to be processed are formed, including: Determining basic expressions related to the multiple frames of images to be processed and determining weight parameters of the basic expressions related to the multiple frames of images to be processed according to the multiple frames of images to be processed, wherein the multiple frames of images to be processed include RGB data and depth data; A plurality of facial data corresponding to the plurality of frames of images to be processed are formed by using base data and weight parameters of basic expressions related to the plurality of frames of images to be processed.

4. The method according to any one of claims 1 to 3, characterized in that: The part of the intraoral model for which no aesthetic design is generated is the intraoral model outside the maxillary anterior teeth area, and the pre-constructed tooth model is the maxillary anterior teeth area tooth model.

5. The method according to any one of claims 1 to 3, characterized in that: Before superimposing the part of the intraoral model without the aesthetic design and the pre-built tooth model onto the neutral facial model with the updated expression state, the method further includes: A tooth model designed for the target user is generated by means of aesthetic design software as the pre-built tooth model.

6. The method according to claim 1, characterized in that Using the basic data of basic expressions, multiple facial data corresponding to the expression states are formed, including: A plurality of facial data corresponding to the expression states are formed by using weight parameters of basic expressions related to expression change images of other users, wherein the expression change images of the other users are obtained by photographing the expression changes of the other users.

7. The method according to claim 2 or 3, characterized in that: Also includes: A three-dimensional animation is generated, wherein the three-dimensional animation includes: an animation with an aesthetic design effect formed by a plurality of frames of the expression images, and / or an animation without an aesthetic design effect formed by a plurality of frames of the images to be processed.

8. The method according to any one of claims 1 to 3, characterized in that: The process of constructing basic data of basic expressions based on the neutral facial model includes: Processing the mesh structure at the lips of the neutral facial model to separate the upper lip and the lower lip of the neutral facial model; identifying feature point information of the neutral facial model after processing; Based on the processed neutral facial model and the feature point information, base data of a plurality of basic expressions are constructed through deformation migration.

9. An image processing device, characterized in that: The device comprises: An acquisition unit is used to acquire relative position relationship information between an intra-oral model and a neutral face model of a target user, including: acquiring a smiling face model, an intra-oral model and a neutral face model of the target user, wherein the smiling face model is a three-dimensional face model scanned when the target user is in a toothy smiling expression state; aligning the intra-oral model to the smiling face model, and aligning the neutral face model to the smiling face model, so as to acquire relative position relationship information between the intra-oral model and the neutral face model, wherein the neutral face model is a three-dimensional face model scanned when the target user is in a non-expression state, and the intra-oral model includes an upper jaw model, a lower jaw model and occlusal relationship data; A forming unit, using base data of basic expressions to form a plurality of facial data corresponding to the expression states, wherein the base data of basic expressions is constructed based on the neutral facial model; An updating unit, configured to update the expression state of the neutral facial model multiple times using the facial data; A superimposition unit is used to superimpose the part of the intraoral model for which no aesthetic design has been generated and the pre-constructed tooth model on the neutral facial model with an updated expression state based on the relative position relationship information, so as to obtain an expression image showing the aesthetic design effect, which includes: identifying feature point information of the neutral facial model, calculating the position change of specific feature points among the feature points, and obtaining a transformation matrix; based on the relative position relationship information and the transformation matrix, obtaining an updated position of the mandibular model in the neutral facial model with an updated expression state; based on the relative position relationship information and the updated position, superimposing the part of the intraoral model for which no aesthetic design has been generated and the pre-constructed tooth model on the neutral facial model with an updated expression state, so as to obtain an expression image showing the aesthetic design effect.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program, and the program is used to implement the image processing method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the image processing method described in any one of claims 1-8.

Images