Video display method, device, electronic device and storage medium

Abstract

The present invention provides a video display method, device, electronic device and storage medium, the method comprising: obtaining the expression changes of a target user to obtain multiple frames of images to be processed; obtaining a first image without an aesthetic design effect and a second image with an aesthetic design effect according to the multiple frames of images to be processed, a pre-constructed tooth model and an intraoral model of the target user; generating a display video consisting of multiple frames of the first image and multiple frames of the second image, or generating a display video consisting of multiple frames of the second image. The display video is used to display an image with an aesthetic restoration effect superimposed thereon to the target user, thereby improving the display effect and user experience of the aesthetic restoration effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video display method, device, electronic equipment and storage medium. Background Art

[0002] As people pursue health and aesthetics, many patients with poor tooth shape hope to beautify their tooth shape through aesthetic restoration. Before formally performing aesthetic restoration on the teeth, the aesthetic restoration effect needs to be shown to the patients in advance, and the aesthetic restoration will only be formally performed on the patients' teeth after they are satisfied.

[0003] The current method of showing the aesthetic restoration effect to patients is to design a tooth model using aesthetic design software, and then superimpose the tooth model on the patient's facial image to simulate the patient's appearance after the aesthetic restoration. However, the aforementioned method can only display the designed tooth model statically, which makes it impossible for patients to fully understand the aesthetic restoration effect, resulting in poor display of the aesthetic restoration effect and poor patient experience. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a video display method, device, electronic device and storage medium to solve the problems of poor display effect of aesthetic restoration effect and poor patient experience in the current way of displaying aesthetic restoration effect to patients.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a video display method, the method comprising:

[0007] Obtain the expression changes of the target user to obtain multiple frames of images to be processed;

[0008] Obtaining a first image without an aesthetic design effect and a second image with an aesthetic design effect according to the plurality of frames of the to-be-processed images, the pre-constructed tooth model and the intraoral model of the target user;

[0009] Generate a display video consisting of multiple frames of the first image and multiple frames of the second image, or generate a display video consisting of multiple frames of the second image;

[0010] In response to an operation instruction for the display video, the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause point are displayed.

[0011] Preferably, obtaining a first image without the aesthetic design effect and a second image with the aesthetic design effect according to the multiple frames of the to-be-processed images, the pre-constructed tooth model and the intraoral model of the target user comprises:

[0012] Based on the image to be processed, performing a specific transformation on the intra-oral model of the target user;

[0013] superimposing the intraoral model after the specific transformation onto the image to be processed to obtain a first image without the aesthetic design effect;

[0014] The part of the intra-oral model after the specific transformation that has not generated the aesthetic design and the pre-built tooth model are superimposed on the image to be processed to obtain a second image with an aesthetic design effect.

[0015] Preferably, in response to the operation instruction for the display video, after displaying the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position, the method further comprises:

[0016] In response to the adjustment instruction, any of the following adjustments are performed on the pre-built tooth model: undo, rollback, reset, color adjustment, posture adjustment, morphology adjustment, and replacement of the tooth model library.

[0017] Preferably, in response to an operation instruction for the display video, displaying the face model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position comprises:

[0018] In response to a pause instruction after playing the display video, a frame image of the display video at the pause point is displayed in a first window, and a facial model and / or an intraoral model and / or a pre-built tooth model of the target user corresponding to the frame number at the pause point is displayed in a second window;

[0019] or,

[0020] In response to a pause instruction and a switch instruction after playing the display video, switching to full-screen display of the face model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the number of frames at the pause position;

[0021] or,

[0022] In response to a pause instruction after playing the display video, displaying a frame image of the display video at the pause point in a third window;

[0023] In response to the switching instruction, the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause point are switched in the third window for display.

[0024] Preferably, the process of displaying the facial model of the target user and / or the intraoral model and / or the pre-built tooth model corresponding to the frame number at the pause position comprises:

[0025] Retrieving and displaying a pre-generated first display model with an aesthetic design effect and / or a second display model without an aesthetic design effect, wherein the first display model is generated at least from a neutral facial model of the target user, the intraoral model and the pre-built tooth model, and the second display model is generated at least from the neutral facial model and the intraoral model;

[0026] or,

[0027] Using the image to be processed corresponding to the number of frames at the pause point and the base data of the basic expression to form facial data;

[0028] Updating the expression state of the neutral facial model of the target user using the facial data;

[0029] Based on the second positional relationship information between the intraoral model and the neutral facial model, the part of the intraoral model for which no aesthetic design is generated and the pre-built tooth model are superimposed on the neutral facial model with an updated expression state to obtain and display a first display model with an aesthetic design effect; and / or, based on the second positional relationship information between the intraoral model and the neutral facial model, the intraoral model is superimposed on the neutral facial model with an updated expression state to obtain and display a second display model without an aesthetic design effect.

[0030] Preferably, based on the image to be processed, a specific transformation is performed on the intraoral model of the target user, including:

[0031] Acquire first positional relationship information between the intra-oral model and the smiling face model of the target user, where the smiling face model is a three-dimensional face model scanned from a state where the target user is in a toothy smiling expression state;

[0032] Mapping the image to be processed to the smiling face model to obtain first mapping relationship information;

[0033] Determining a first corresponding relationship between the image to be processed and the intraoral model according to the first position relationship information and the first mapping relationship information;

[0034] The intra-oral model is specifically transformed using the first corresponding relationship.

[0035] Preferably, based on the image to be processed, a specific transformation is performed on the intraoral model of the target user, including:

[0036] Acquire second positional relationship information between the intra-oral model of the target user and the neutral facial model, where the neutral facial model is a three-dimensional facial model scanned when the target user is in a state of no expression;

[0037] Mapping the image to be processed to the neutral face model to obtain second mapping relationship information;

[0038] Determining a second corresponding relationship between the image to be processed and the intraoral model according to the second position relationship information and the second mapping relationship information;

[0039] The intra-oral model is specifically transformed using the second corresponding relationship.

[0040] Preferably, before superimposing the part of the intraoral model after the specific transformation for which no aesthetic design is generated and the pre-built tooth model onto the image to be processed, the method further comprises:

[0041] Eliminate the lip line area of ​​the image to be processed.

[0042] Preferably, before obtaining the first image without the aesthetic design effect and the second image with the aesthetic design effect, the method further includes:

[0043] A tooth model designed for the target user is generated as a pre-built tooth model through aesthetic design software.

[0044] A second aspect of an embodiment of the present invention discloses a video display device, the device comprising:

[0045] An acquisition unit, used to acquire the expression changes of the target user to obtain multiple frames of images to be processed;

[0046] A processing unit, configured to obtain a first image without an aesthetic design effect and a second image with an aesthetic design effect according to a plurality of frames of the to-be-processed images, a pre-constructed tooth model and an intraoral model of the target user;

[0047] A generating unit, configured to generate a display video consisting of a plurality of frames of the first image and a plurality of frames of the second image, or to generate a display video consisting of a plurality of frames of the second image;

[0048] The display unit is used to display the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause point in response to an operation instruction for the displayed video.

[0049] A third aspect of an embodiment of the present invention discloses an electronic device, comprising: a processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; and the memory is used to store a program, wherein the program is used to implement the video display method disclosed in the first aspect of the embodiment of the present invention.

[0050] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the video display method disclosed in the first aspect of the embodiment of the present invention.

[0051] Based on the above-mentioned embodiment of the present invention, a video display method, device, electronic device and storage medium are provided. The method is: obtaining the expression changes of the target user to obtain multiple frames of images to be processed; obtaining a first image without aesthetic design effects and a second image with aesthetic design effects based on the multiple frames of images to be processed, a pre-built tooth model and an intraoral model of the target user; generating a display video consisting of multiple frames of first images and multiple frames of second images, or generating a display video consisting of multiple frames of second images. In this solution, the expression changes of the target user are obtained to obtain multiple frames of images to be processed. Multiple frames of images to be processed are used to generate a display video with at least an aesthetic design effect. The display video is used to display images with superimposed aesthetic restoration effects to the target user, thereby improving the display effect and user experience of the aesthetic restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A flowchart of a video display method provided by an embodiment of the present invention;

[0054] Figure 2 An example diagram of an image to be processed provided by an embodiment of the present invention;

[0055] Figure 3 Another example diagram of an image to be processed provided by an embodiment of the present invention;

[0056] Figure 4 An example diagram of an intraoral model provided by an embodiment of the present invention;

[0057] Figure 5 An example diagram of a smiling face model provided by an embodiment of the present invention;

[0058] Figure 6 An example diagram of a neutral face model provided by an embodiment of the present invention;

[0059] Figure 7 An example diagram of a lip line area in an image to be processed provided by an embodiment of the present invention;

[0060] Figure 8 An example diagram of a tooth model provided by an embodiment of the present invention;

[0061] Fig. 9 An example diagram of a video display provided by an embodiment of the present invention;

[0062] Fig.10 An example diagram of basic expressions provided by an embodiment of the present invention;

[0063] Fig.11 Another example diagram of a basic expression provided by an embodiment of the present invention;

[0064] Fig.12 A structural block diagram of a video display device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0067] As can be seen from the background technology, as people pursue health and aesthetics, many patients with bad tooth shapes hope to beautify their tooth shapes through aesthetic restoration, such as through digital smile design (DSD) to beautify their tooth shapes and show a brighter smile, thereby improving the patient's self-confidence. Before formally performing aesthetic restoration of teeth, it is necessary to show the aesthetic restoration effect to the patient in advance, and only after the patient is satisfied can the patient's teeth be formally restored aesthetically.

[0068] At present, the way to show the effect of aesthetic restoration to patients is to design a tooth model using aesthetic design software (such as DSD aesthetic design software), and then superimpose the tooth model on the patient's facial image to simulate the patient's appearance after the aesthetic restoration. In other words, at present, the patient's tooth design data (or tooth simulation design data) is simply superimposed on the patient's two-dimensional facial image to simulate the patient's appearance after the aesthetic restoration.

[0069] However, the existing method of displaying the aesthetic restoration effect can only display the designed tooth model statically, which results in that the patient cannot fully understand the aesthetic restoration effect, the display effect of the aesthetic restoration effect is poor and the patient experience is poor.

[0070] In order to solve the above problems, this solution proposes a video display method, device, electronic device and storage medium, which obtains the expression changes of the target user to obtain multiple frames of images to be processed. The multiple frames of images to be processed are used to generate a display video with at least an aesthetic design effect. The image with the superimposed aesthetic restoration effect is displayed to the target user by displaying the video, thereby improving the display effect and user experience of the aesthetic restoration effect.

[0071] In other words, this solution brings a series of facial expressions and movements of patients who need aesthetic restoration into the display of aesthetic restoration effects, and shows patients images of aesthetic restoration effects superimposed under a series of facial expressions and movements, thereby fully reflecting the effect picture of the patient after completing the aesthetic restoration.

[0072] The present solution is described in detail below through various embodiments.

[0073] It should be noted that the subsequent explanation of this plan will involve scanning the target user's three-dimensional facial model and other personal information; before scanning the target user's three-dimensional facial model, this plan has informed the target user and obtained the target user's authorization to collect personal information, that is, this plan collects the target user's three-dimensional facial model and other personal information on the premise of complying with relevant laws and regulations.

[0074] See also Figure 1 , shows a flow chart of a video display method provided by an embodiment of the present invention, the video processing method comprising:

[0075] Step S101: Obtain the expression changes of the target user to obtain multiple frames of images to be processed.

[0076] It should be noted that the target users are patients who need aesthetic dental restoration.

[0077] In the process of implementing step S101, the expression changes of the target user are obtained (including shooting or retrieving), so as to collect multiple frames of images to be processed (expression data) that can be used for expression tracking calculation. The images to be processed contain RGB data and depth data (Depth data), where RGB refers to red, green, and blue (red, green, blue).

[0078] For example: capture the target user's facial expression changes, and collect information such as Figure 2 and Figure 3 The image to be processed is shown in Figure 2 and Figure 3 The images to be processed are images taken by the target user with different expressions.

[0079] Step S102: obtaining a first image without the aesthetic design effect and a second image with the aesthetic design effect according to the multiple frames of images to be processed, the pre-constructed tooth model and the intra-oral model of the target user.

[0080] It should be noted that the intraoral model, smiling face model and neutral face model of the target user are obtained by pre-scanning, wherein the intraoral model includes the maxillary model, the mandibular model and the occlusal relationship data, the smiling face model is a three-dimensional face model scanned when the target user is in a toothy smiling expression state, and the neutral face model is a three-dimensional face model scanned when the target user is in a non-expression state;

[0081] The following is an explanation of how to scan and obtain the intraoral model, smiling face model, and neutral face model:

[0082] The target user's mouth is scanned by a scanning device to obtain an intraoral model of the target user; specifically, the upper jaw and lower jaw in the target user's mouth are scanned by a scanning device to obtain the upper jaw model and lower jaw model of the target user respectively, and the occlusal relationship data is scanned.

[0083] It should be noted that in the process of scanning the target user's occlusion data, the target user's upper and lower jaws need to be aligned. The scanned occlusion data can be used as a reference for the "change in the occlusion position of the upper and lower jaws" in the subsequent expression change processing.

[0084] For example: Scan to get Figure 4 The intraoral model of the target user is shown, and the intraoral model includes a maxillary model, a mandibular model and occlusal relationship data.

[0085] When the target user is in a toothy smiling expression state, the three-dimensional facial model of the target user is scanned by a scanning device to obtain the smiling facial model of the target user.

[0086] It should be noted that, during the process of scanning the target user's smiling facial model, the target user's teeth need to maintain a normal occlusion relationship (the upper jaw and the lower jaw need to be aligned with each other), that is, the occlusion relationship corresponds to the occlusion relationship when the intraoral model is obtained.

[0087] For example, when the target user's teeth maintain a normal occlusion relationship and the target user is in a toothy smile, the scanned image is as follows: Figure 5 The smiling face model of the target user is shown.

[0088] The target user is in a state of no expression (or the target user's mouth is naturally closed), and the three-dimensional facial model of the target user is scanned by a scanning device to obtain a neutral facial model of the target user.

[0089] For example, the target user is in a normal state with his mouth closed, and the scan results are as follows: Figure 6 The neutral facial model of the target user is shown.

[0090] The smiling face model, intraoral model and neutral face model of the target user can be scanned and acquired in the above manner.

[0091] It should be noted that the data collected by different scanning devices are usually not in an aligned state, and the data collected multiple times by the same scanning device are usually not in an aligned state; therefore, after scanning and obtaining the smiling face model, intraoral model and neutral face model of the target user, it is necessary to align the smiling face model, intraoral model and neutral face model of the target user.

[0092] In a specific implementation, the intra-oral model of the target user is aligned to the smiling face model of the target user to obtain first position relationship information between the intra-oral model and the smiling face model;

[0093] The neutral facial model of the target user is then aligned to the smiling facial model of the target user, and combined with the first positional relationship information, second positional relationship information between the intra-oral model and the neutral facial model can be obtained.

[0094] That is to say, the target user's intra-oral model is first aligned to the smiling face model, and then the neutral face model is aligned to the smiling face model. This indirectly aligns the intra-oral model to the neutral face model, thereby obtaining the second positional relationship information between the intra-oral model and the neutral face model.

[0095] It should be noted that the alignment operations of "aligning the intraoral model to the smiling facial model" and "aligning the neutral facial model to the smiling facial model" can be implemented using a grid matching algorithm, that is, the intraoral model is aligned to the smiling facial model, and the neutral facial model is aligned to the smiling facial model through a grid matching algorithm.

[0096] In some embodiments, the above-mentioned grid matching algorithm includes a grid coarse matching algorithm, a grid fine matching algorithm, etc. For example, the grid matching algorithm is a grid matching algorithm such as PCA and ICP.

[0097] In the process of specifically implementing step S102, based on the image to be processed, a specific transformation is performed on the intraoral model of the target user; the intraoral model after the specific transformation is superimposed on the image to be processed to obtain a first image without aesthetic design effect; the part of the intraoral model after the specific transformation for which no aesthetic design is generated and the pre-built tooth model are superimposed on the image to be processed to obtain a second image with aesthetic design effect.

[0098] It should be noted that the first image without aesthetic design effects can also be obtained from the image to be processed without superimposing the intraoral model, that is, there is no need to perform additional processing on the tooth part of the image to be processed, and the tooth part captured in the image to be processed can be directly presented.

[0099] It can be seen from the above content that the first positional relationship information between the intraoral model and the smiling facial model is known, and the second positional relationship information between the intraoral model and the neutral facial model is known; therefore, after capturing the image to be processed, for each frame of the image to be processed, the frame of the image to be processed is mapped to the smiling facial model or the neutral facial model, and the correspondence between the frame of the image to be processed and the intraoral model can be obtained. The correspondence is recorded as a specific transformation from the intraoral model to the frame of the image to be processed (such as RT transformation), and the intraoral model can be subjected to a specific transformation (such as RT transformation) through the corresponding relationship.

[0100] Specifically, under the premise that the first position relationship information and the second position relationship information are known, the first position relationship information or the second position relationship information can be used to perform a specific transformation on the intra-oral model. The following specifically explains how to perform a specific transformation on the intra-oral model.

[0101] In some embodiments, the first method of performing a specific transformation on the intra-oral model is: obtaining first position relationship information between the intra-oral model and the smiling face model of the target user; for each frame of the image to be processed (RGB data and Depth data), mapping the frame of the image to be processed to the smiling face model to obtain first mapping relationship information; determining a first corresponding relationship between the frame of the image to be processed and the intra-oral model according to the first position relationship information and the first mapping relationship information; performing a specific transformation on the intra-oral model using the first corresponding relationship;

[0102] The first corresponding relationship is recorded as a specific transformation (such as RT transformation) from the intra-oral model to the frame of the image to be processed. The first corresponding relationship can be used to perform a specific transformation (such as RT transformation) on the intra-oral model.

[0103] In some embodiments, the second method of performing a specific transformation on the intra-oral model is: obtaining second position relationship information between the intra-oral model of the target user and the neutral facial model; for each frame of the image to be processed (RGB data and Depth data), mapping the frame of the image to be processed to the neutral facial model to obtain second mapping relationship information; determining a second corresponding relationship between the frame of the image to be processed and the intra-oral model according to the second position relationship information and the second mapping relationship information; performing a specific transformation on the intra-oral model using the second corresponding relationship;

[0104] The second corresponding relationship is recorded as a specific transformation (such as RT transformation) from the intra-oral model to the frame of image to be processed, and the intra-oral model can be subjected to a specific transformation (such as RT transformation) through the second corresponding relationship.

[0105] It should be noted that before superimposing the part of the intraoral model after the specific transformation that does not generate the aesthetic design and the pre-built tooth model onto the image to be processed, the lip line area of ​​the image to be processed is eliminated.

[0106] Specifically, the method of removing the lip line area of ​​the image to be processed is: for each frame of the image to be processed, the lip line area of ​​the frame of the image to be processed is removed, and the lip line boundary of the frame of the image to be processed from which the lip line area is removed is morphologically processed (equivalent to repairing the lip morphology).

[0107] For each frame of image to be processed (RGB data and Depth data) collected, the lip line area in the frame of image to be processed is automatically identified and removed through the specified image AI algorithm; then, algorithms such as boundary smoothing are used to perform morphological processing on the lip line boundary of the frame of image to be processed with the lip line area removed.

[0108] For example: Using the specified image AI algorithm, automatically identify Figure 7 The lip line area in the image to be processed is shown, and the lip line area in the image to be processed is removed.

[0109] For each frame of the image to be processed, the intraoral model after a specific transformation is superimposed on the frame of the image to be processed, so as to obtain a corresponding frame of the first image without the aesthetic design effect, and the first image is an image without the superimposed aesthetic restoration effect.

[0110] It should be noted that, for each frame of image to be processed, the "intraoral model after a specific transformation" superimposed on the frame of image to be processed specifically refers to: the intraoral model after a specific transformation is performed through the frame of image to be processed and the first positional relationship information (or the second positional relationship information).

[0111] In some embodiments, a tooth model of the target user after aesthetic restoration (referred to as a pre-constructed tooth model) is pre-constructed. Specifically, a tooth model designed for the target user is generated in advance through aesthetic design software as the pre-constructed tooth model.

[0112] For example: through DSD aesthetic design software, generate Figure 8 The DSD aesthetic design model of the maxillary anterior teeth area of ​​the target user is shown, and the DSD aesthetic design model is a pre-built tooth model.

[0113] It should be noted that the pre-constructed tooth model may be a DSD anterior teeth aesthetic veneer model, or may be other types of aesthetic design models, and the specific circumstances of the pre-constructed tooth model are not limited herein.

[0114] For each frame of image to be processed, the part of the intraoral model after a specific transformation that has not generated the aesthetic design and the pre-built tooth model are superimposed on the frame of image to be processed, so as to obtain a corresponding frame of second image with aesthetic design effect, which is an image superimposed with aesthetic restoration effect.

[0115] In some embodiments, the portion of the intraoral model for which no aesthetic design is generated is the intraoral model outside the maxillary anterior teeth area, and the pre-constructed tooth model is the maxillary anterior teeth area tooth model.

[0116] Step S103: Generate a display video consisting of multiple frames of first images and multiple frames of second images, or generate a display video consisting of multiple frames of second images.

[0117] Through the above step S102, a plurality of first image frames without superimposed aesthetic restoration effects and a plurality of second image frames with superimposed aesthetic restoration effects can be obtained;

[0118] For example, multiple frames of first images without DSD effect and multiple frames of second images with DSD effect can be obtained.

[0119] In the specific implementation of step S103, a display video is generated consisting of multiple frames of first images and multiple frames of second images obtained in the above step S102, or a display video is generated consisting of multiple frames of second images obtained in the above step S102.

[0120] That is to say, the generated display video contains multiple frames of first images without DSD effect and multiple frames of second images with DSD effect (simultaneously showing images with and without superimposed aesthetic restoration effect), or, the generated display video only contains multiple frames of second images with DSD effect (only showing images with superimposed aesthetic restoration effect).

[0121] In some embodiments, after the demonstration video is generated, the demonstration video is played to demonstrate the aesthetic restoration effect.

[0122] In the case where the display video is composed of multiple frames of first images and multiple frames of second images, a specific method of playing the display video is: playing the multiple frames of first images and multiple frames of second images in the first playback area and the second playback area in the display video respectively.

[0123] That is to say, in the process of playing a display video composed of multiple frames of first images and multiple frames of second images, the first image without superimposed aesthetic restoration effect (such as without DSD effect) is played frame by frame in the first playback area in the display video. At the same time, the second image with superimposed aesthetic restoration effect (such as with DSD effect) is played frame by frame in the second playback area in the display video.

[0124] From a visual perspective, the demonstration video composed of multiple frames of first images and multiple frames of second images simultaneously displays multiple frames of first images without superimposed aesthetic restoration effects and multiple frames of second images with superimposed aesthetic restoration effects, so that the target users can clearly compare the images before and after aesthetic restoration.

[0125] For example, during the playback of a display video composed of multiple frames of first images and multiple frames of second images, one of the first and second frames of the display video is Fig. 9 As shown, Fig. 9 The left image is the first image without DSD effect ( Fig. 9 The left area of ​​​​the first playback area), Fig. 9 The right image in the figure is the second image with DSD effect ( Fig. 9 The area to the right of is the second play area).

[0126] Step S104: In response to an operation instruction for displaying the video, a facial model and / or an intraoral model and / or a pre-built tooth model of the target user corresponding to the frame number at the pause point is displayed.

[0127] In the specific implementation of step S104, after the display video is generated, the target user can trigger an operation instruction for the display video to view the corresponding three-dimensional model;

[0128] For example, during the playback of a demonstration video, the target user can trigger corresponding operation instructions for the demonstration video to switch to view the corresponding three-dimensional model.

[0129] That is to say, during the playback of the demonstration video, the target user or relevant personnel can pause the playback of the demonstration video, and then switch to the three-dimensional state to browse the three-dimensional model by triggering corresponding operation instructions.

[0130] It should be noted that the operation instruction for displaying the video includes a pause instruction after playing, or the operation instruction for displaying the video includes a pause instruction after playing and a switch instruction;

[0131] In some embodiments, the specific method of responding to the operation instruction for displaying the video is:

[0132] In response to a pause instruction after playing the display video, a frame image of the display video at the pause point is displayed in a first window, and a facial model and / or an intraoral model and / or a pre-built tooth model of the target user corresponding to the frame number at the pause point is displayed in a second window;

[0133] or,

[0134] In response to a pause instruction and a switch instruction after playing the display video, switch to full-screen display of the target user's facial model and / or intraoral model and / or pre-built tooth model corresponding to the frame number at the pause position;

[0135] or,

[0136] In response to a pause instruction after playing the display video, a frame image of the display video at the pause point is displayed in a third window; in response to a switch instruction, the facial model and / or intraoral model and / or pre-built dental model of the target user corresponding to the frame number at the pause point is switched to display in the third window.

[0137] That is to say, when the target user triggers a pause command after playback for the display video, a frame image of the display video at the pause point (that is, one of the frame images constituting the display video) is displayed in the first window, and the facial model and / or intraoral model and / or pre-built dental model of the target user corresponding to the frame number at the pause point is displayed in the second window. In this way, the target user can view the image in the first window and the corresponding three-dimensional model in the second window without switching.

[0138] The target user triggers a pause command and a switch command after playback for the displayed video, first pausing the displayed video, and then switching to full-screen display of the target user's facial model and / or intraoral model and / or pre-built tooth model corresponding to the frame number at the pause point.

[0139] The target user first triggers a playback and pause command for the display video, at which point the frame image of the display video at the pause point is displayed in the third window. When the target user wants to view the three-dimensional model, he or she can trigger a switching command, at which point the facial model and / or intraoral model and / or pre-built dental model of the target user corresponding to the frame number at the pause point is switched to display in the third window.

[0140] The specific display method for the content of the target user's facial model and / or intraoral model and / or pre-built tooth model corresponding to the frame number at the above display pause point is as follows:

[0141] Retrieving and displaying a pre-generated first display model with an aesthetic design effect and / or a second display model without an aesthetic design effect, wherein the first display model is generated at least from a neutral facial model, an intraoral model, and a pre-built tooth model of a target user, and the second display model is generated at least from a neutral facial model and an intraoral model;

[0142] or,

[0143] Facial data is formed using the image to be processed corresponding to the number of frames at the pause point and the base data of the basic expression; the facial data is used to update the expression state of the neutral facial model of the target user; based on the second positional relationship information between the intraoral model and the neutral facial model, the part of the intraoral model for which no aesthetic design is generated and the pre-built tooth model are superimposed on the neutral facial model with the updated expression state to obtain and display a first display model with an aesthetic design effect; and / or, based on the second positional relationship information between the intraoral model and the neutral facial model, the intraoral model is superimposed on the neutral facial model with the updated expression state to obtain and display a second display model without the aesthetic design effect.

[0144] That is, the first display model and the second display model may be generated in advance, or the first display model and the second display model may be generated using the image to be processed;

[0145] If the first display model and the second display model are generated in advance, the first display model with the aesthetic design effect and / or the second display model without the aesthetic design effect may be directly retrieved and displayed.

[0146] If the first display model and the second display model are not generated in advance, it is necessary to update the expression state of the neutral facial model of the target user using the facial data;

[0147] Then, based on the second positional relationship information between the intraoral model and the neutral facial model, the part of the intraoral model for which no aesthetic design has been generated and the pre-built tooth model are superimposed on the neutral facial model with an updated expression state, so as to generate and display a first display model with an aesthetic design effect; and / or, based on the second positional relationship information between the intraoral model and the neutral facial model, the intraoral model is superimposed on the neutral facial model with an updated expression state, so as to generate and display a second display model without an aesthetic design effect.

[0148] The target user can browse the three-dimensional facial model with expressions with and / or without aesthetic restoration effects by pausing to view the first display model and / or the second display model.

[0149] It should be noted that the basic data of the basic expressions mentioned above are constructed based on the neutral facial model of the target user. The specific method of constructing the basic data of the basic expressions based on the neutral facial model of the target user is as follows:

[0150] Processing the mesh structure at the lips of the neutral facial model to separate the upper lip and the lower lip of the neutral facial model; identifying feature point information of the processed neutral facial model (the neutral facial model after the upper lip and the lower lip are separated); and constructing base data of multiple basic expressions through deformation migration based on the processed neutral facial model and the feature point information.

[0151] Among them, the basic expression can also be called the expression base. The base data of multiple basic expressions are constructed by deformation migration of the expression base template. The expression base template library contains at least: single expressions such as left eye blink, right eye blink, open mouth, and upturned mouth. The expression base template is selected from the expression base template library, and the deformation migration is used to construct the base data of multiple basic expressions corresponding to the target user. Fig.10 This is an example of the basic expression "open mouth". Fig.11 This is an example of the basic expression "tilted lips".

[0152] The constructed base data of the basic expressions of the target user are numbered, for example, A1, A2 and A3 are used to represent the base data of different basic expressions respectively; the constructed base data of each basic expression has the same grid topology as the neutral facial model.

[0153] Based on the above content about basic expressions, the specific method of forming facial data using the base data of the images to be processed and the basic expressions is: according to multiple frames of images to be processed, determining the basic expressions related to the multiple frames of images to be processed, and determining the weight parameters of the basic expressions related to the multiple frames of images to be processed, the multiple frames of images to be processed include RGB data and depth data; using the base data and weight parameters of the basic expressions related to the multiple frames of images to be processed, forming multiple facial data corresponding to the multiple frames of images to be processed.

[0154] It should be noted that, for each frame of image to be processed, the base data of the basic expressions are interpolated to obtain the weight parameters corresponding to the base data of each basic expression related to the frame of image to be processed; using the base data of these basic expressions and their corresponding weight parameters, the facial data corresponding to the frame of image to be processed can be formed.

[0155] Specifically, the facial data A corresponding to the image to be processed can be constructed by multiplying the basic data of a finite number of basic expressions in pre-constructed basic expressions by a certain weight parameter, for example, A=c1*A1+c5*A5+c8*A8, A1, A5 and A8 are the basic data of the basic expressions related to the frame of image to be processed, and c1, c5 and c8 are the weight parameters corresponding to A1, A5 and A8.

[0156] The facial data A formed in the above manner has the same mesh topology as the neutral facial model, and the facial data A formed is the expression state of the three-dimensional facial model.

[0157] It should be noted that after displaying the facial model and / or intraoral model and / or pre-built dental model of the target user corresponding to the frame number at the pause point, the target user can modify the displayed pre-built dental model. For example, if the target user is not satisfied, the displayed pre-built dental model can be modified.

[0158] In some embodiments, in response to the adjustment instruction, any of the following adjustments are made to the pre-built tooth model: undo, rollback, reset, color adjustment, posture adjustment, morphology adjustment, and replacement of the tooth model library.

[0159] That is, the target user can trigger adjustment instructions to undo, roll back, reset, adjust the color, adjust the posture, adjust the shape, and change any item in the tooth model library of the pre-built tooth model.

[0160] In an embodiment of the present invention, the expression changes of the target user are obtained to obtain multiple frames of images to be processed. The multiple frames of images to be processed are used to generate a display video with at least an aesthetic design effect. The display video is used to display an image with an aesthetic restoration effect superimposed to the target user, thereby improving the display effect of the aesthetic restoration effect and the user experience.

[0161] Corresponding to a video display method provided by the above embodiment of the present invention, see Fig.12 , the embodiment of the present invention also provides a structural block diagram of a video display device, the video display device includes: an acquisition unit 100, a processing unit 200, a generation unit 300 and a display unit 400;

[0162] The acquisition unit 100 is used to acquire the expression changes of the target user to obtain multiple frames of images to be processed.

[0163] The processing unit 200 is used to obtain a first image without aesthetic design effects and a second image with aesthetic design effects according to multiple frames of images to be processed, a pre-built tooth model and an intra-oral model of a target user.

[0164] The generating unit 300 is used to generate a display video consisting of a plurality of frames of first images and a plurality of frames of second images, or to generate a display video consisting of a plurality of frames of second images.

[0165] The display unit 400 is used to display the facial model and / or intraoral model and / or pre-built tooth model of the target user corresponding to the frame number at the pause point in response to an operation instruction for displaying the video.

[0166] In an embodiment of the present invention, the expression changes of the target user are obtained to obtain multiple frames of images to be processed. The multiple frames of images to be processed are used to generate a display video with at least an aesthetic design effect. The display video is used to display an image with an aesthetic restoration effect superimposed to the target user, thereby improving the display effect of the aesthetic restoration effect and the user experience.

[0167] Preferably, combined Fig.12 As shown in the figure, the processing unit 200 includes a transformation subunit, a first superposition subunit and a second superposition subunit, and the execution principle of each subunit is as follows:

[0168] The transformation subunit is used to perform specific transformation on the intra-oral model of the target user based on the image to be processed.

[0169] In some embodiments, the transformation subunit is specifically used to: obtain first position relationship information between the intra-oral model of the target user and the smiling facial model, where the smiling facial model is a three-dimensional facial model scanned when the target user is in a toothy smiling expression state; map the image to be processed to the smiling facial model to obtain first mapping relationship information; determine a first corresponding relationship between the image to be processed and the intra-oral model based on the first position relationship information and the first mapping relationship information; and use the first corresponding relationship to perform a specific transformation on the intra-oral model.

[0170] In some other embodiments, the transformation subunit is specifically used to: obtain second positional relationship information between the intraoral model of the target user and the neutral facial model, where the neutral facial model is a three-dimensional facial model scanned when the target user is in an expressionless state; map the image to be processed to the neutral facial model to obtain second mapping relationship information; determine a second correspondence relationship between the image to be processed and the intraoral model based on the second positional relationship information and the second mapping relationship information; and use the second correspondence relationship to perform a specific transformation on the intraoral model.

[0171] The first superimposing subunit is used to superimpose the intraoral model after a specific transformation onto the image to be processed to obtain a first image without the aesthetic design effect.

[0172] The second superposition subunit is used to superimpose the part of the intraoral model after the specific transformation that has not generated the aesthetic design and the pre-built tooth model on the image to be processed to obtain a second image with an aesthetic design effect.

[0173] Accordingly, the processing unit 200 further includes:

[0174] The culling subunit is used to cull the lip line area of ​​the image to be processed.

[0175] Preferably, combined Fig.12 The video display device also includes:

[0176] The adjustment unit is used to make any of the following adjustments to the pre-built tooth model in response to the adjustment instruction: undo, rollback, reset, color adjustment, posture adjustment, morphology adjustment and replacement of the tooth model library.

[0177] Preferably, combined Fig.12 The display unit 400 is specifically used for:

[0178] In response to a pause instruction after playing the display video, a frame image of the display video at the pause point is displayed in a first window, and a facial model and / or an intraoral model and / or a pre-built tooth model of the target user corresponding to the frame number at the pause point is displayed in a second window;

[0179] or,

[0180] In response to a pause instruction and a switch instruction after playing the display video, switch to full-screen display of the target user's facial model and / or intraoral model and / or pre-built tooth model corresponding to the frame number at the pause position;

[0181] or,

[0182] In response to a pause instruction after playing the display video, displaying a frame image of the display video at the pause point in the third window;

[0183] In response to the switching instruction, the facial model and / or intraoral model and / or pre-built tooth model of the target user corresponding to the frame number at the pause point is switched to be displayed in the third window.

[0184] Accordingly, the process of the display unit 400 displaying the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position includes:

[0185] Retrieving and displaying a pre-generated first display model with an aesthetic design effect and / or a second display model without an aesthetic design effect, wherein the first display model is generated at least from a neutral facial model, an intraoral model, and a pre-built tooth model of a target user, and the second display model is generated at least from a neutral facial model and an intraoral model;

[0186] or,

[0187] The facial data is formed by using the image to be processed corresponding to the frame number at the pause point and the base data of the basic expression;

[0188] Updating the expression state of the neutral facial model of the target user using the facial data;

[0189] Based on the second positional relationship information between the intraoral model and the neutral facial model, the part of the intraoral model for which no aesthetic design has been generated and the pre-built tooth model are superimposed on the neutral facial model with an updated expression state to obtain and display a first display model with an aesthetic design effect; and / or, based on the second positional relationship information between the intraoral model and the neutral facial model, the intraoral model is superimposed on the neutral facial model with an updated expression state to obtain and display a second display model without an aesthetic design effect.

[0190] Preferably, combined Fig.12 The video display device also includes:

[0191] The design unit is used to generate a tooth model designed for a target user as a pre-built tooth model through aesthetic design software.

[0192] Preferably, an embodiment of the present invention further provides an electronic device, comprising: a processor and a memory, the processor and the memory being connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; the memory is used to store the program, and the program is used to implement the video display method disclosed in the above method embodiment.

[0193] Preferably, an embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the video display method disclosed in the above method embodiment.

[0194] In summary, the embodiments of the present invention provide a video display method, device, electronic device, and storage medium, which obtain the expression changes of the target user to obtain multiple frames of images to be processed. The multiple frames of images to be processed are used to generate a display video with at least an aesthetic design effect. The image with the superimposed aesthetic restoration effect is displayed to the target user by displaying the video, thereby improving the display effect and user experience of the aesthetic restoration effect.

[0195] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0196] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0197] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video display method, characterized in that: The method comprises: Obtain the expression changes of the target user to obtain multiple frames of images to be processed; Obtaining a first image without an aesthetic design effect and a second image with an aesthetic design effect according to the plurality of frames of the to-be-processed images, the pre-constructed tooth model and the intraoral model of the target user; Generate a display video consisting of multiple frames of the first image and multiple frames of the second image, or generate a display video consisting of multiple frames of the second image; In response to an operation instruction for the display video, displaying the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position; The process of displaying the facial model of the target user and / or the intraoral model and / or the pre-built tooth model corresponding to the number of frames at the pause position comprises: using the base data of the image to be processed and the basic expression corresponding to the number of frames at the pause position to form facial data; using the facial data to update the expression state of the neutral facial model of the target user; based on the second positional relationship information between the intraoral model and the neutral facial model, superimposing the part of the intraoral model for which no aesthetic design is generated and the pre-built tooth model on the neutral facial model with the updated expression state, to obtain and display a first display model with an aesthetic design effect; And / or, based on the second positional relationship information between the intraoral model and the neutral facial model, the intraoral model is superimposed on the neutral facial model with updated expression state to obtain and display a second display model without aesthetic design effect; the first display model is generated by at least the neutral facial model of the target user, the intraoral model and the pre-constructed tooth model, and the second display model is generated by at least the neutral facial model and the intraoral model; the pre-constructed tooth model is a tooth model after aesthetic restoration; the neutral facial model is a three-dimensional facial model scanned when the target user is in an expressionless state.

2. The method according to claim 1, characterized in that: Obtaining a first image without an aesthetic design effect and a second image with an aesthetic design effect according to the plurality of frames of the to-be-processed images, the pre-constructed tooth model, and the intraoral model of the target user, comprising: Based on the image to be processed, performing a specific transformation on the intra-oral model of the target user; superimposing the intraoral model after the specific transformation onto the image to be processed to obtain a first image without the aesthetic design effect; The part of the intra-oral model after the specific transformation that has not generated the aesthetic design and the pre-built tooth model are superimposed on the image to be processed to obtain a second image with an aesthetic design effect.

3. The method according to claim 1, characterized in that After the face model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position are displayed in response to the operation instruction for the display video, the method further includes: In response to the adjustment instruction, any of the following adjustments are performed on the pre-built tooth model: undo, rollback, reset, color adjustment, posture adjustment, morphology adjustment, and replacement of the tooth model library.

4. The method according to claim 1, characterized in that In response to the operation instruction for the display video, the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position are displayed, comprising: In response to a pause instruction after playing the display video, a frame image of the display video at the pause point is displayed in a first window, and a facial model and / or an intraoral model and / or a pre-built tooth model of the target user corresponding to the frame number at the pause point is displayed in a second window; or, In response to a pause instruction and a switch instruction after playing the display video, switching to full-screen display of the face model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the number of frames at the pause position; or, In response to a pause instruction after playing the display video, displaying a frame image of the display video at the pause point in a third window; In response to the switching instruction, the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause point are switched in the third window for display.

5. The method according to claim 2, characterized in that: Based on the image to be processed, a specific transformation is performed on the intraoral model of the target user, including: Acquire first positional relationship information between the intra-oral model and the smiling face model of the target user, where the smiling face model is a three-dimensional face model scanned from a state where the target user is in a toothy smiling expression state; Mapping the image to be processed to the smiling face model to obtain first mapping relationship information; Determining a first corresponding relationship between the image to be processed and the intraoral model according to the first position relationship information and the first mapping relationship information; The intra-oral model is specifically transformed using the first corresponding relationship.

6. The method according to claim 2, characterized in that Based on the image to be processed, a specific transformation is performed on the intraoral model of the target user, including: Acquire second positional relationship information between the intra-oral model of the target user and the neutral facial model; Mapping the image to be processed to the neutral face model to obtain second mapping relationship information; Determining a second corresponding relationship between the image to be processed and the intraoral model according to the second position relationship information and the second mapping relationship information; The intra-oral model is specifically transformed using the second corresponding relationship.

7. The method according to claim 2, characterized in that: Before superimposing the part of the intraoral model after the specific transformation that does not generate the aesthetic design and the pre-built tooth model onto the image to be processed, the method further includes: Eliminate the lip line area of ​​the image to be processed.

8. The method according to any one of claims 1 to 7, characterized in that: Before obtaining the first image without the aesthetic design effect and the second image with the aesthetic design effect, the method further includes: A tooth model designed for the target user is generated as a pre-built tooth model through aesthetic design software.

9. A video display device, characterized in that: The device comprises: An acquisition unit, used to acquire the expression changes of the target user to obtain multiple frames of images to be processed; A processing unit, configured to obtain a first image without an aesthetic design effect and a second image with an aesthetic design effect according to a plurality of frames of the to-be-processed images, a pre-constructed tooth model and an intraoral model of the target user; A generating unit, configured to generate a display video consisting of a plurality of frames of the first image and a plurality of frames of the second image, or to generate a display video consisting of a plurality of frames of the second image; A display unit, configured to display the facial model and / or the intraoral model and / or the pre-built tooth model of the target user corresponding to the frame number at the pause position in response to an operation instruction for the display video; The process of displaying the facial model of the target user and / or the intraoral model and / or the pre-built tooth model corresponding to the number of frames at the pause position by the display unit comprises: forming facial data by using the image to be processed and the base data of the basic expression corresponding to the number of frames at the pause position; updating the expression state of the neutral facial model of the target user by using the facial data; and superimposing the part of the intraoral model that does not generate the aesthetic design and the pre-built tooth model on the neutral facial model with the updated expression state based on the second position relationship information between the intraoral model and the neutral facial model, so as to obtain and display the first display with the aesthetic design effect. Model; and / or, based on the second positional relationship information between the intraoral model and the neutral facial model, superimposing the intraoral model onto the neutral facial model with updated expression state, to obtain and display a second display model without aesthetic design effect; the first display model is generated by at least the neutral facial model of the target user, the intraoral model and the pre-constructed tooth model, and the second display model is generated by at least the neutral facial model and the intraoral model; the pre-constructed tooth model is a tooth model after aesthetic restoration; the neutral facial model is a three-dimensional facial model scanned when the target user is in an expressionless state.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program, and the program is used to implement the video display method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the video display method described in any one of claims 1-8.

Images