DATA GENERATION DEVICE, DATA GENERATION METHOD, AND DATA GENERATION PROGRAM

Abstract

To provide a technique capable of easily and accurately simulating the face after a dental treatment.SOLUTION: A data generation device 10 comprises: a moving image data interface 13 and a three-dimensional data interface 14 which receive inputs of moving image data obtained by imaging an object person in such a state that a partial tooth row in upper and lower tooth rows is exposed and three-dimensional data of the upper and lower tooth rows of the object person; and a calculation device 11 which generates simulation moving image data indicating the face of the object person on which the simulation result of the dental treatment is reflected on the basis of the partial tooth row included in each of a plurality of frame images constituting the moving image data and the three-dimensional data of the upper and lower tooth rows.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a data generating device, a data generating method, and a data generating program for generating simulation video data showing a subject's face reflecting a simulation result of dental treatment. [Background technology]

[0002] Conventionally, in dental treatments such as orthodontics, simulations are performed to show changes in the patient's facial appearance. Such simulations are important from the viewpoints of predicting orthodontic results and explaining them to patients, and there is a demand for simulations in which the patient makes facial expressions, especially smiles, in a more natural state.

[0003] Furthermore, advances in computer image processing technology have made it possible to create a virtual three-dimensional surface model of the patient's face, and the patient's surface model can be used to confirm changes in facial shape or jaw movement due to orthodontic treatment. For example, Patent Document 1 discloses a dental computed tomography device capable of creating a virtual three-dimensional surface model of the patient's face. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-176852 A Summary of the Invention [Problem to be solved by the invention]

[0005] According to the dental computed tomography apparatus disclosed in Patent Document 1, it is possible to create a virtual three-dimensional surface model of a patient's face, but since it is necessary to set up an imaging station including multiple components such as a color camera, a laser device, and an illumination device, the preparation is troublesome. Therefore, there is a demand for a technology that can easily and accurately simulate a patient's face.

[0006] The present disclosure has been made to solve such problems, and has an object to provide a technique that enables a face after dental treatment to be simulated easily and accurately. [Means for solving the problem]

[0007] According to one example of the present disclosure, there is provided a data generating device for generating simulation video data showing a subject's face reflecting a simulation result of dental treatment. The data generating device includes an input unit for inputting video data obtained by photographing the subject with some of the upper and lower dentitions exposed and three-dimensional data of the subject's upper and lower dentitions, and a calculation unit for generating simulation video data showing the subject's face reflecting a simulation result of dental treatment based on the some of the dentitions included in each of a plurality of frame images constituting the video data and the three-dimensional data of the upper and lower dentitions. The calculation unit generates at least one rendering image including two-dimensional upper and lower teeth corresponding to a portion of the teeth included in each of the multiple frame images based on three-dimensional data of the subject's upper and lower teeth in which a dental treatment simulation is performed on the upper and lower teeth and the simulation results are reflected, and generates simulation video data by converting the image of the portion of the teeth before the simulation results included in each of the multiple frame images are reflected into an image of the portion of the upper and lower teeth in which the simulation results included in the at least one rendering image are reflected.

[0008] According to one example of the present disclosure, there is provided a data generation method for generating simulation video data showing a subject's face reflecting a result of a computer-generated simulation of dental treatment. The data generation method includes a step of inputting video data obtained by photographing the subject with a part of the upper and lower dentition exposed and three-dimensional data of the upper and lower dentition of the subject, and a step of generating simulation video data showing the subject's face reflecting the result of a dental treatment simulation based on the part of the dentition included in each of a plurality of frame images constituting the video data and the three-dimensional data of the upper and lower dentition. The generating step includes a step of generating at least one rendering image including two-dimensional upper and lower teeth corresponding to a portion of the teeth included in each of the plurality of frame images based on three-dimensional data of the subject's upper and lower teeth on which a dental treatment simulation has been performed and the simulation results have been reflected, and a step of generating simulation video data by converting an image of the portion of the teeth before the simulation results included in each of the plurality of frame images are reflected into an image of the portion of the upper and lower teeth on which the simulation results included in the at least one rendering image are reflected.

[0009] According to one example of the present disclosure, there is provided a data generation program for generating simulation video data showing a subject's face reflecting a simulation result of dental treatment. The data generation program causes a computer to execute the steps of inputting video data obtained by photographing a subject with some of the upper and lower dentitions exposed and three-dimensional data of the subject's upper and lower dentitions, and generating simulation video data showing the subject's face reflecting a simulation result of dental treatment based on the some of the dentitions included in each of a plurality of frame images constituting the video data and the three-dimensional data of the upper and lower dentitions. The generating step includes a step of generating at least one rendering image including two-dimensional upper and lower teeth corresponding to a portion of the teeth included in each of the plurality of frame images based on three-dimensional data of the subject's upper and lower teeth on which a dental treatment simulation has been performed and the simulation results have been reflected, and a step of generating simulation video data by converting an image of the portion of the teeth before the simulation results included in each of the plurality of frame images are reflected into an image of the portion of the upper and lower teeth on which the simulation results included in the at least one rendering image are reflected. Effect of the Invention

[0010] According to the present disclosure, a user can obtain simulation video data showing the subject's face reflecting the results of a dental treatment simulation by using video data obtained by photographing the subject with some of the upper and lower dentition exposed and three-dimensional data of the subject's upper and lower dentition, thereby enabling a user to easily and accurately simulate the subject's face after dental treatment. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an application example of a data generation system and a data generation device according to a first embodiment. [Diagram 2] 1 is a block diagram showing a hardware configuration of a data generating device according to a first embodiment. [Diagram 3] 4 is a flowchart illustrating an example of a data generation process executed by the data generating device according to the first embodiment. [Figure 4] FIG. 11 is a diagram for explaining the processing content of STEP 1 in the data generation processing. [Diagram 5] FIG. 11 is a diagram for explaining the processing content of STEP 5 in the data generation processing. [Figure 6]11 is a diagram for explaining the processing contents of STEP 7 and STEP 8 in the data generation processing. FIG. [Figure 7] 11 is a diagram for explaining the processing contents of STEP 7 and STEP 8 in the data generation processing. FIG. [Figure 8] 11 is a diagram for explaining the processing contents of STEP 7 and STEP 8 in the data generation processing. FIG. [Figure 9] 11 is a diagram for explaining the processing contents of STEP 7 and STEP 8 in the data generation processing. FIG. [Figure 10] 11 is a diagram for explaining the processing contents of STEP 7 and STEP 8 in the data generation processing. FIG. [Figure 11] FIG. 11 is a diagram for explaining the processing content of STEP 9 in the data generation processing. [Figure 12] FIG. 11 is a diagram for explaining the processing content of STEP 10 in the data generation processing. [Figure 13] FIG. 11 is a diagram for explaining the processing content of STEP 11 in the data generation processing. [Figure 14] 11 is a flowchart illustrating an example of a data generation process executed by a data generating device according to a second embodiment. [Figure 15] 13 is a flowchart illustrating an example of a data generation process executed by a data generating device according to a third embodiment. [Figure 16] FIG. 11 is a diagram for explaining an overview of supervised learning executed by a data generating device according to a third embodiment. [Figure 17] FIG. 11 is a diagram for explaining input and output of supervised learning in a data generating device according to a third embodiment. [Figure 18] FIG. 11 is a diagram for explaining input and output of supervised learning in a data generating device according to a third embodiment. [Figure 19] 13 is a flowchart illustrating an example of a data generation process executed by a data generating device according to a fourth embodiment. [Figure 20]13 is a flowchart illustrating an example of a data generation process executed by a data generating device according to a fifth embodiment. [Figure 21] FIG. 23 is a diagram illustrating an application example of a data generation system and a data generation device according to a sixth embodiment. [Figure 22] FIG. 23 is a diagram for explaining input and output of supervised learning in a data generating device according to a seventh embodiment. [Diagram 23] FIG. 23 is a diagram showing an example of input data for supervised learning in the data generating device according to the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment 1> A first embodiment of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference characters and the description thereof will not be repeated.

[0013] [Example of application] An application example of the data generating system 1 and the data generating device 10 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an application example of the data generating system 1 and the data generating device 10 according to the first embodiment.

[0014] For example, a simulation showing a change in a patient's facial appearance may be performed in association with a simulation of dental treatments such as whitening, all-ceramic crowns, laminate veneers, metal bonds, cleaning, and vestments, which are included in orthodontic or aesthetic treatments. Such a simulation is important from the viewpoint of predicting orthodontic treatments and explaining them to patients, and it is required to perform a simulation of a patient's facial expression, especially a smile, in a state closer to natural. Therefore, the data generating system 1 and the data generating device 10 according to the first embodiment are configured to easily and accurately perform a simulation of the subject's face after dental treatment using video data obtained by photographing the subject with some of the upper and lower teeth exposed and three-dimensional data of the upper and lower teeth of the subject.

[0015] Specifically, as shown in FIG. 1, the data generation system 1 includes a camera 20 capable of capturing moving images, a three-dimensional scanner 30 capable of acquiring three-dimensional data of an object, and a data generation device 10.

[0016] The camera 20 may be any camera such as a digital camera or a video camera that can capture a moving image of a subject (for example, an RGB camera). In the first embodiment, a monocular camera is exemplified as the camera 20. The moving image data acquired by the camera 20 is composed of a plurality of frame images arranged in time series. A frame image is one still image showing a two-dimensional object to be captured as seen from the viewpoint of the camera 20 directed toward the object to be captured (for example, the face of the subject). For example, when the frame rate of the camera 20 is set to 24 fps, the moving image data acquired by the camera 20 includes 24 frame images per second. When it is desired to check jaw movement with higher time resolution, a camera with a high frame rate such as 120 fps or 240 fps may be used as the camera 20.

[0017] The three-dimensional scanner 30 is a so-called intraoral scanner (IOS: Intra Oral Scanner) capable of optically imaging the inside of the oral cavity of a subject by a confocal method, a triangulation method, or the like. Specifically, the three-dimensional scanner 30 scans the oral cavity, and obtains, as three-dimensional data, position information (coordinates of each axis in the vertical, horizontal, and height directions) of each of the point clouds (multiple points) indicating the surface of the subject (object) to be photographed in the oral cavity, using an optical sensor or the like. That is, the three-dimensional data is position data including position information of each of the points constituting the surface of the object to be scanned (for example, the upper and lower teeth in the oral cavity) placed in a certain coordinate space. When the upper and lower teeth in the oral cavity are scanned by the three-dimensional scanner 30, three-dimensional data including position information of each of the points indicating the surface of the upper and lower teeth is obtained. Such three-dimensional data makes it possible to specify the shape of the object (upper and lower teeth) placed in the coordinate space. Hereinafter, a collection of three-dimensional data capable of specifying the shape of the object (upper and lower teeth) placed in the coordinate space is also referred to as "shape data". Moreover, data capable of identifying the position of an object (upper and lower teeth rows) placed in the coordinate space is also referred to as "position data."

[0018] The user uses the camera 20 to capture a video of the subject with some of the upper and lower dentition exposed. Furthermore, the user can obtain three-dimensional data of the upper and lower dentition in the oral cavity by scanning the oral cavity of the subject using the three-dimensional scanner 30. The video data acquired by the camera 20 and the three-dimensional data acquired by the three-dimensional scanner 30 are input to the data generating device 10. Note that the data generating device 10 is not limited to the case where video data obtained by photographing the subject is input, and may also be input with a plurality of still images obtained by continuously photographing the subject in chronological order.

[0019] The "user" includes practitioners (such as doctors) or assistants (such as dental assistants, dental technicians, nurses, etc.) in various fields such as dentistry, oral surgery, orthopedics, plastic surgery, and cosmetic surgery. The "subject" includes patients of dentistry, oral surgery, orthopedics, plastic surgery, and cosmetic surgery. The "upper and lower dentition" includes both the upper and lower dentition in the oral cavity, or either the upper and lower dentition. That is, the "upper and lower dentition" includes at least one of the upper and lower dentition. The "part of the upper and lower dentition" includes both the part of the upper dentition and the part of the lower dentition, or either the part of the upper dentition and the part of the lower dentition. That is, the "part of the upper and lower dentition" includes at least one of the part of the upper dentition and the part of the lower dentition.

[0020] The data generating device 10 generates simulation video data (video data consisting of frame images Ma, Mb, Mc, and Md shown in Figure 1) showing changes in the patient's facial appearance when dental treatment is performed, based on video data of the subject acquired by the camera 20 and three-dimensional data of the subject's upper and lower dentition acquired by the three-dimensional scanner 30.

[0021] This allows the user to obtain simulation video data showing the subject's face reflecting the results of the dental treatment simulation by using video data obtained by photographing the subject with some of the upper and lower dentition exposed and three-dimensional data of the subject's upper and lower dentition, thereby enabling the user to easily and accurately simulate the subject's face after dental treatment.

[0022] [Hardware configuration of the identification device] A hardware configuration of the data generating device 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing a hardware configuration of the data generating device 10 according to the first embodiment. The data generating device 10 may be realized, for example, by a general-purpose computer or a computer dedicated to the data generating system 1.

[0023] As shown in FIG. 2, the data generating device 10 includes, as its main hardware elements, a computing device 11, a memory device 12, a video data interface 13, a three-dimensional data interface 14, a display interface 15, a peripheral device interface 16, a media reading device 17, and a communication device 18.

[0024] The arithmetic device 11 is an arithmetic entity (computer) that executes various processes by executing various programs, and is an example of a "arithmetic unit." The arithmetic device 11 is configured with a processor such as a CPU (central processing unit) or an MPU (micro-processing unit). The processor, which is an example of the arithmetic device 11, has a function of executing various processes by executing programs, but some or all of these functions may be implemented using a dedicated hardware circuit such as an ASIC (application specific integrated circuit) or an FPGA (field-programmable gate array). The "processor" is not limited to a processor in the narrow sense that executes processes using a stored program method such as a CPU or an MPU, but may include a hardwired circuit such as an ASIC or an FPGA. Therefore, the "processor," which is an example of the arithmetic device 11, may be read as a processing circuitry in which processing is defined in advance by computer-readable code and / or a hardwired circuit. The arithmetic device 11 may be configured with one chip or multiple chips. Furthermore, some functions of the arithmetic device 11 may be provided in a server device (for example, a cloud-type server device) not shown.

[0025] The storage device 12 includes a volatile storage area (e.g., a working area) that temporarily stores program code, work memory, and the like when the arithmetic device 11 executes an arbitrary program. For example, the storage device 12 includes volatile memory such as DRAM (dynamic random access memory) and SRAM (static random access memory), or non-volatile memory such as ROM (Read Only Memory) and flash memory. Furthermore, the storage device 12 may be an SSD (solid state drive) or HDD (hard disk drive), and in this case, the arithmetic device 11 may include volatile memory such as DRAM and SRAM, or non-volatile memory such as ROM and flash memory.

[0026] In the first embodiment, an example has been shown in which the volatile storage area and the nonvolatile storage area are included in the same storage device 12, but the volatile storage area and the nonvolatile storage area may be included in different storage units. For example, the arithmetic device 11 may include a volatile storage area, and the storage device 12 may include a nonvolatile storage area. The data generating device 10 may include a microcomputer including the arithmetic device 11 and the storage device 12.

[0027] The storage device 12 stores a data generation program 100. The data generation program 100 describes the contents of a data generation process for the computing device 11 to generate simulation moving image data showing the face of a subject reflecting a simulation result of dental treatment based on moving image data of the subject and three-dimensional data of the upper and lower teeth of the subject.

[0028] The video data interface 13 is an interface for connecting the camera 20, and is an example of an "input unit." The video data interface 13 realizes input and output of data (for example, video data) between the data generating device 10 and the camera 20. The data generating device 10 and the camera 20 are connected via a wired connection using a cable, or wirelessly (WiFi, Bluetooth (registered trademark), etc.).

[0029] The three-dimensional data interface 14 is an interface for connecting the three-dimensional scanner 30, and is an example of an "input unit." The three-dimensional data interface 14 realizes input and output of data (e.g., three-dimensional data) between the data generating device 10 and the three-dimensional scanner 30. The data generating device 10 and the three-dimensional scanner 30 are connected via a wired connection using a cable, or wirelessly (WiFi, Bluetooth (registered trademark), etc.).

[0030] The display interface 15 is an interface for connecting to the display 40. The display interface 15 realizes input and output of data between the data generating device 10 and the display 40.

[0031] The peripheral device interface 16 is an interface for connecting peripheral devices such as a keyboard 51 and a mouse 52. The peripheral device interface 16 realizes input and output of data between the data generating device 10 and the peripheral devices.

[0032] The media reading device 17 reads various data stored in a removable disk 60, which is a storage medium, and writes various data to the removable disk 60. For example, the media reading device 17 may obtain the data generating program 100 from the removable disk 60, or may write simulation video data generated by the arithmetic device 11 to the removable disk 60.

[0033] The communication device 18 transmits and receives data to and from an external device via wired or wireless communication. For example, the communication device 18 may transmit simulation video data generated by the arithmetic device 11 to the external device.

[0034] [Data generation process] The data generation process executed by the data generating device 10 according to the first embodiment will be described with reference to Figs. 3 to 13. In the data generation process according to the first embodiment shown in Figs. 3 to 13, an example will be described in which simulation video data showing changes in a patient's facial appearance when dental treatment is performed is generated by pattern matching using each frame image included in the video data. Fig. 3 is a flowchart for explaining an example of the data generation process executed by the data generating device 10 according to the first embodiment. Each STEP (hereinafter, indicated by "S") shown in Fig. 3 is realized by the calculation device 11 of the data generating device 10 executing the data generation program 100.

[0035] As shown in FIG. 3, after video data of a subject captured by a camera 20 is input, the data generating device 10 extracts a plurality of frame images constituting the video data from the video data (S1).

[0036] For example, FIG. 4 is a diagram for explaining the processing contents of STEP 1 in the data generation processing. As shown in FIG. 4, when the camera 20 captures a video of a subject smiling with some of the rows of teeth exposed, the video data acquired by the camera 20 includes a plurality of frame images A, B, C, and D showing the subject smiling with some of the rows of teeth exposed. For example, the frame image A includes a portion of the rows of teeth with the upper and lower rows of teeth closed, and the frame images B to D include a portion of the rows of teeth with the upper and lower rows of teeth open. The frame images A to D show how the upper and lower rows of teeth change from a closed state to an open state. Note that while the subject is being video-captured, the position (viewpoint) of the camera 20 and the position of the subject are not necessarily fixed, so that the upper and lower rows of teeth shown by the frame images A to D acquired in time series may look different from each other from the camera position. The data generation device 10 extracts a plurality of frame images showing the state in which such a subject gradually opens the upper and lower rows of teeth from a closed state from the video data acquired by the camera 20.

[0037] Returning to FIG. 3, in steps S2 to S4, the data generating device 10 executes pre-processing for generating simulation moving image data showing the face of the subject reflecting the results of the dental treatment simulation.

[0038] Specifically, when at least one of the multiple frame images has distortion due to the lens of the camera 20, the data generating device 10 corrects the distortion in at least one of the frame images (S2). By performing such distortion correction, the data generating device 10 can improve the accuracy of pattern matching between the shape of the upper and lower teeth included in the frame images and the shape of the upper and lower teeth realized based on the three-dimensional data acquired by the three-dimensional scanner 30, which will be described later.

[0039] The data generating device 10 detects the positions of the subject (for example, the subject's face) and the camera 20 (S3). The data generating device 10 can improve the accuracy of pattern matching by using the positions of the subject and the camera 20 in pattern matching, which will be described later, and can also reduce the processing load of pattern matching and shorten the processing time of pattern matching. The data generating device 10 may obtain the positions of the subject and the camera 20 at the start of shooting by the camera 20 or during shooting, and store them in the storage device 12. In the process of S3, the data generating device 10 may search and obtain the positions of the subject and the camera 20 stored by the camera 20, or may search and obtain the positions of the subject and the camera 20 stored by the storage device 12.

[0040] The data generating device 10 detects the distance between the subject to be photographed (for example, the subject's face) and the camera 20 (S4). The data generating device 10 uses the distance between the subject to be photographed and the camera 20 in pattern matching, which will be described later, to reduce the processing load for pattern matching and shorten the processing time for pattern matching. The data generating device 10 may obtain the distance between the subject to be photographed and the camera 20 at the start of photographing by the camera 20 or during photographing, and store it in the storage device 12. In the process of S3, the data generating device 10 may obtain the distance by searching for the distance stored by the camera 20, or may obtain the distance by searching for the distance stored by the storage device 12.

[0041] The data generating device 10 may perform pattern matching, which will be described later, based on at least one of the positional relationship between the subject and the camera 20 and the distance between the subject and the camera 20 acquired in S3.

[0042] The data generating device 10 extracts upper and lower dental row images corresponding to the upper and lower dental row from each frame image extracted in S1 (S5).

[0043] For example, FIG. 5 is a diagram for explaining the processing contents of STEP 5 in the data generation processing. As shown in FIG. 5, the data generation device 10 extracts upper and lower teeth row images corresponding to the upper and lower teeth row from each of the frame images A, B, C, D. For example, the data generation device 10 compares each of the frame images A, B, C, D with a template image showing the shape of the upper and lower teeth row stored in advance, and extracts the upper and lower teeth row images from each of the frame images A, B, C, D by pattern matching. The data generation device 10 may extract the upper and lower teeth row images from each of the frame images A, B, C, D by pattern matching that compares the edge shapes of the upper and lower teeth row, not limited to pattern matching that compares the shapes of the upper and lower teeth row. The "upper and lower teeth row image" is an image that includes both the upper and lower teeth row in the oral cavity, or either the upper and lower teeth row. Furthermore, the "upper and lower teeth row image" is an image that includes a part of the upper and lower teeth row.

[0044] The data generating device 10 extracts upper and lower teeth row image a, which shows the upper and lower teeth row of the subject, from the part of the subject's face included in frame image A. The data generating device 10 extracts upper and lower teeth row image b, which shows the upper and lower teeth row of the subject, from the part of the subject's face included in frame image B. The data generating device 10 extracts upper and lower teeth row image c, which shows the upper and lower teeth row of the subject, from the part of the subject's face included in frame image C. The data generating device 10 extracts upper and lower teeth row image d, which shows the upper and lower teeth row of the subject, from the part of the subject's face included in frame image D.

[0045] 3, after the three-dimensional data of the subject's upper and lower dentition acquired by the three-dimensional scanner 30 is input, the data generating device 10 performs a simulation of dental treatment on the upper and lower dentition (S6). Specifically, the data generating device 10 converts the input three-dimensional data of the upper and lower dentition, thereby updating the input three-dimensional data of the upper and lower dentition before the dental treatment to three-dimensional data of the upper and lower dentition after the dental treatment.

[0046] The data generating device 10 generates multiple rendering images including two-dimensional upper and lower teeth viewed from multiple directions based on three-dimensional data after simulation of the upper and lower teeth that have undergone dental treatment. A "rendering image" is an image generated by processing or editing certain data, and in the first embodiment, it is a two-dimensional image generated by processing or editing the three-dimensional data after simulation.

[0047] As described above, the three-dimensional data acquired by the three-dimensional scanner 30 includes position information of each of the point clouds representing the surfaces of the upper and lower teeth of the subject, and therefore the three-dimensional data after the simulation also includes position information of each of the point clouds representing the surfaces of the upper and lower teeth on which dental treatment has been performed. The data generating device 10 can generate a rendering image showing the two-dimensional upper and lower teeth as seen from a certain viewpoint by imaging the point clouds representing the surfaces of the upper and lower teeth after dental treatment using the three-dimensional data after the simulation. Furthermore, the data generating device 10 can generate multiple rendering images including the two-dimensional upper and lower teeth as seen from multiple directions by changing the viewpoint in multiple directions.

[0048] The data generating device 10 generates a plurality of rendering images including two-dimensional upper and lower teeth rows viewed from multiple directions based on the three-dimensional data after simulation, and roughly pattern matches the upper and lower teeth row images included in each frame image extracted in S5 with the upper and lower teeth row images included in each of the plurality of rendering images (S7), and further performs detailed pattern matching between the upper and lower teeth row images included in each frame image extracted in S5 with the upper and lower teeth row images included in each of the plurality of rendering images (S8).

[0049] For example, Figs. 6 to 10 are diagrams for explaining the processing contents of STEP 7 and STEP 8 in the data generation processing. As shown in Fig. 6, the data generation device 10 can generate a plurality of rendering images including two-dimensional upper and lower teeth rows viewed from multiple directions in the horizontal and vertical directions by virtually viewing an object (upper and lower teeth rows) placed in a coordinate space that can be specified based on three-dimensional data while changing the viewpoint (position of the camera 20) and the angle of view. The data generation device 10 then searches for a rendering image of an upper and lower teeth row image that appears to match the upper and lower teeth row image included in each frame image by pattern matching, and generates (extracts) a rendering image corresponding to the upper and lower teeth row image included in each frame image. Note that "search" means to search for a rendering image that matches the upper and lower teeth row image included in the frame image from among the plurality of rendering images by trying pattern matching between each of the plurality of rendering images and the upper and lower teeth row image included in the frame image once or multiple times. When the data generation device 10 generates a rendering image corresponding to the upper and lower teeth row image included in the frame image, it stores position data of the object (upper and lower teeth row) at the time when the rendering image was generated. Alternatively, the data generating device 10 stores the change in position data of the object (upper and lower dental rows) (the amount of movement of the object) when generating each rendering image corresponding to each frame image.

[0050] An example of generation of a rendering image will be described with reference to Fig. 7 and Fig. 8. As shown in Fig. 7, the data generating device 10 assumes that the viewpoint when generating the rendering image is moved stepwise in the horizontal direction from a state facing the front of the subject's face (upper and lower teeth rows), and generates each rendering image including two-dimensional upper and lower teeth rows viewed from each viewpoint in the horizontal direction.

[0051] Specifically, the data generating device 10 generates a rendering image TC including two-dimensional upper and lower teeth rows when the subject's face (upper and lower teeth rows) is viewed from the front based on the three-dimensional data after the simulation. The data generating device 10 generates a rendering image TL including two-dimensional upper and lower teeth rows when the subject's face (upper and lower teeth rows) is viewed with the viewpoint slightly tilted left in the horizontal direction from the front of the face (upper and lower teeth rows) as an origin (for example, when the viewpoint is tilted 10 degrees left in the horizontal direction from the front of the face) based on the three-dimensional data after the simulation. The data generating device 10 generates a rendering image TR including two-dimensional upper and lower teeth rows when the subject's face (upper and lower teeth rows) is viewed with the viewpoint slightly tilted right in the horizontal direction from the front of the face (upper and lower teeth rows) as an origin (for example, when the viewpoint is tilted 10 degrees right in the horizontal direction from the front of the face).

[0052] As shown in Figure 8, the data generating device 10 assumes that the viewpoint when generating the rendering image is moved stepwise in the vertical direction from a state facing the subject's face (upper and lower teeth), and generates each rendering image including two-dimensional upper and lower teeth viewed from each viewpoint in the vertical direction.

[0053] Specifically, the data generating device 10 generates a rendering image TC including two-dimensional upper and lower teeth rows when the subject's face (upper and lower teeth rows) is viewed from the front based on the three-dimensional data after the simulation. The data generating device 10 generates a rendering image TD including two-dimensional upper and lower teeth rows when the subject's face (upper and lower teeth rows) is viewed with the viewpoint tilted slightly downward in the vertical direction from the front of the face (upper and lower teeth rows) as an origin (for example, when the viewpoint is tilted vertically downward by 10 degrees from the front as an origin) based on the three-dimensional data after the simulation. The data generating device 10 generates a rendering image TU including two-dimensional upper and lower teeth rows when the subject's face (upper and lower teeth rows) is viewed with the viewpoint tilted slightly upward in the vertical direction from the front of the face (upper and lower teeth rows) as an origin (for example, when the viewpoint is tilted vertically upward by 10 degrees from the front as an origin).

[0054] Although not shown, the data generating device 10 generates a rendering image including two-dimensional upper and lower teeth rows when viewed with the viewpoint tilted slightly downward in the vertical direction (for example, when the viewpoint is tilted 10 degrees downward in the vertical direction from the front as a starting point) using the rendering image TL as a reference, and a rendering image including two-dimensional upper and lower teeth rows when viewed with the viewpoint tilted slightly upward in the vertical direction (for example, when the viewpoint is tilted 10 degrees upward in the vertical direction from the front as a starting point). The data generating device 10 generates a rendering image including two-dimensional upper and lower teeth rows when viewed with the viewpoint tilted slightly downward in the vertical direction (for example, when the viewpoint is tilted 10 degrees downward in the vertical direction from the front as a starting point) using the rendering image TR as a reference, and a rendering image including two-dimensional upper and lower teeth rows when viewed with the viewpoint tilted slightly upward in the vertical direction (for example, when the viewpoint is tilted 10 degrees upward in the vertical direction from the front as a starting point).

[0055] In this way, the data generating device 10 can generate a plurality of rendering images with the viewpoint tilted stepwise in the horizontal direction, and a plurality of rendering images with the viewpoint tilted stepwise in the vertical direction. Note that in the data generating device 10, the angle of tilt in the horizontal or vertical direction starting from the front of the subject's face (upper and lower teeth) is preset, and the more precisely the angle is set, the more rendering images the data generating device 10 can generate based on viewpoints from multiple directions.

[0056] In S7, the data generating device 10 uses a rendering image tilted at a rough angle (e.g., 3 degree intervals) in the horizontal or vertical direction to roughly search for a rendering image including an upper and lower dental row image that matches the upper and lower dental row image included in each frame image by pattern matching. Then, in S8, the data generating device 10 generates a rendering image tilted at a fine angle (e.g., 1 degree intervals) in the horizontal or vertical direction based on the rendering image generated (extracted) in S7, and searches in detail by pattern matching for a rendering image including an upper and lower dental row image that matches the upper and lower dental row image included in each frame image.

[0057] An example of pattern matching will be described with reference to Figures 9 and 10. As shown in Figure 9, the data generating device 10 performs pattern matching in the horizontal direction between the upper and lower teeth row images extracted in S5 based on each frame image constituting the video data and each rendering image including the two-dimensional upper and lower teeth row seen from each viewpoint in the horizontal direction, and extracts the viewpoint of the camera 20 in the horizontal direction of the upper and lower teeth row images extracted from each frame image.

[0058] For example, the data generating device 10 performs pattern matching in the horizontal direction between upper and lower dental row image a extracted from frame image A showing the subject with the upper and lower dental rows closed and multiple rendering images TC, TL, TR including two-dimensional upper and lower dental rows viewed from each viewpoint in the horizontal direction generated based on three-dimensional data after simulation, and extracts a rendering image corresponding to upper and lower dental row image a from the multiple rendering images TC, TL, TR.

[0059] Specifically, the data generating device 10 performs pattern matching in the horizontal direction between the shape (appearance) of the upper and lower teeth included in the upper and lower teeth image a and the shape (appearance) of the upper and lower teeth included in each of the multiple rendering images TC, TL, and TR, thereby extracting a rendering image including the shape of the upper and lower teeth most similar in the horizontal direction to the shape of the upper and lower teeth included in the upper and lower teeth image a from among the multiple rendering images TC, TL, and TR. At this time, the data generating device 10 may perform pattern matching in the horizontal direction based on the characteristics of the teeth. For example, the data generating device 10 may perform pattern matching in the horizontal direction for shapes near the cutting edge of the incisor, near the occlusal surface of the canine or molar, etc. In the example of FIG. 9, the data generating device 10 extracts the rendering image TC as a rendering image that matches the upper and lower teeth image a in the horizontal direction. This allows the data generating device 10 to extract the viewpoint of the camera 20 in the horizontal direction of the upper and lower teeth image a. For the other frame images B to D, the data generating device 10 can extract the viewpoint of the camera 20 in the horizontal direction of the upper and lower tooth row images included in each of the frame images B to D in the same manner as for the frame image A.

[0060] As shown in FIG. 10, the data generating device 10 performs pattern matching in the vertical direction between the upper and lower dental row images extracted in S5 based on each frame image constituting the video data and each rendering image including the two-dimensional upper and lower dental row viewed from each viewpoint in the vertical direction, and extracts the viewpoint of the camera 20 in the vertical direction of the upper and lower dental row images extracted from each frame image.

[0061] For example, the data generating device 10 performs pattern matching in the vertical direction between upper and lower dental row image a extracted from frame image A showing the subject with the upper and lower dental rows closed and multiple rendering images TC, TD, and TU including two-dimensional upper and lower dental rows seen from each viewpoint in the vertical direction generated based on three-dimensional data after simulation, and extracts a rendering image corresponding to upper and lower dental row image a from the multiple rendering images TC, TD, and TU.

[0062] Specifically, the data generating device 10 performs pattern matching in the vertical direction between the shape (appearance) of the upper and lower teeth included in the upper and lower teeth image a and the shape (appearance) of the upper and lower teeth included in each of the multiple rendering images TC, TD, and TU, thereby extracting a rendering image including the shape of the upper and lower teeth most similar in the vertical direction to the shape of the upper and lower teeth included in the upper and lower teeth image a from among the multiple rendering images TC, TD, and TU. At this time, the data generating device 10 may perform pattern matching in the vertical direction based on the characteristics of the teeth. For example, the data generating device 10 may perform pattern matching in the vertical direction on the shapes of the vicinity of the cutting edge of the incisor, the vicinity of the occlusal surface of the canine or molar, and the like. In the example of FIG. 10, the data generating device 10 extracts the rendering image TC as a rendering image that matches the upper and lower teeth image a in the vertical direction. This allows the data generating device 10 to extract the viewpoint of the camera 20 in the vertical direction of the upper and lower teeth image a. For the other frame images B to D, the data generating device 10 can extract the viewpoint of the camera 20 in the vertical direction of the upper and lower tooth row images included in each of the frame images B to D in the same manner as for the frame image A.

[0063] In addition, the data generating device 10 is not limited to generating multiple rendering images based on viewpoints in multiple directions while performing pattern matching, and may generate multiple rendering images based on viewpoints in multiple directions in advance before performing pattern matching.

[0064] In S7 and S8, when performing pattern matching between the upper and lower teeth images extracted from each frame image and the multiple rendering images, the data generating device 10 may use the respective positions of the subject (e.g., the subject's face) and the camera 20 acquired in S3. For example, the data generating device 10 may enlarge or reduce the upper and lower teeth images included in the multiple rendering images in accordance with the size of the upper and lower teeth images based on the respective positions of the subject (e.g., the subject's face) and the camera 20. This allows the data generating device 10 to improve the accuracy of pattern matching by image recognition, and further reduce the processing load for pattern matching and the processing time for pattern matching.

[0065] Furthermore, since the data generating device 10 corrects distortions in the frame images in advance in S2, the upper and lower dentition images extracted from each frame image to be subjected to pattern matching can be made clearer. This allows the data generating device 10 to improve the accuracy of pattern matching by image recognition.

[0066] Returning to Figure 3, the data generating device 10 generates rendering images corresponding to the upper and lower dental row images contained in each frame image based on the viewpoint of the camera 20 in the horizontal and vertical directions of the upper and lower dental row images contained in each frame image extracted by pattern matching in S7 and S8 (S9).

[0067] For example, Fig. 11 is a diagram for explaining the process content of STEP 9 in the data generation process. As shown in Fig. 11, the data generation device 10 generates rendering images Ra to Rd seen from each of the viewpoints in the horizontal and vertical directions based on the viewpoints in each of the frame images A to D extracted in S7 and S8.

[0068] Specifically, the data generating device 10 generates a rendering image Ra in which the upper and lower teeth rows are closed, in accordance with the upper and lower teeth row image a included in the frame image A. The data generating device 10 generates a rendering image Rb in which the upper and lower teeth rows in the closed state shown in the rendering image Ra are changed to a slightly open state, in accordance with the upper and lower teeth row image b included in the frame image B. The data generating device 10 generates a rendering image Rc in which the upper and lower teeth rows in the closed state shown in the rendering image Ra are changed to a more open state than the rendering image Rb, in accordance with the upper and lower teeth row image c included in the frame image C. The data generating device 10 generates a rendering image Rd in which the upper and lower teeth rows in the closed state shown in the rendering image Ra are changed to a more open state than the rendering image Rc, in accordance with the upper and lower teeth row image d included in the frame image D.

[0069] Returning to FIG. 3, the data generating device 10 cuts out the portions corresponding to the upper and lower tooth row images extracted from each frame image in S5 from each of the generated rendering images Ra, Rb, Rc, and Rd (S10).

[0070] For example, Fig. 12 is a diagram for explaining the processing content of STEP 10 in the data generation processing. As shown in Fig. 12, the data generation device 10 cuts out a portion corresponding to the upper and lower teeth image extracted from frame image A from the rendering image Ra. The data generation device 10 cuts out a portion corresponding to the upper and lower teeth image extracted from frame image B from the rendering image Rb. The data generation device 10 cuts out a portion corresponding to the upper and lower teeth image extracted from frame image C from the rendering image Rc. The data generation device 10 cuts out a portion corresponding to the upper and lower teeth image extracted from frame image D from the rendering image Rd.

[0071] Returning to Figure 3, the data generating device 10 converts the portion of each frame image showing the upper and lower dentition images into the portion corresponding to the upper and lower dentition images cut out from the rendering image in S10, thereby generating simulation video data showing the subject's face reflecting the simulation results, and stores the data in the storage device 12 (S11).

[0072] For example, Fig. 13 is a diagram for explaining the processing contents of STEP 11 in the data generation processing. As shown in Fig. 13, the data generation device 10 converts the part of the frame image A showing the upper and lower teeth image a into a part corresponding to the upper and lower teeth image cut out from the rendering image Ra to generate a frame image Ma reflecting the simulation result. At this time, the data generation device 10 processes the frame image Ma reflecting the simulation result so as to smoothly connect the fusion part between the converted upper and lower teeth part and the unconverted part other than the upper and lower teeth. Note that "conversion" includes overlaying the part of the frame image showing the upper and lower teeth image on the part corresponding to the upper and lower teeth image cut out from the rendering image, or replacing the part of the frame image showing the upper and lower teeth image with the part corresponding to the upper and lower teeth image cut out from the rendering image.

[0073] Similarly, the data generating device 10 converts the portion of the frame image B showing the upper and lower teeth image b into a portion corresponding to the upper and lower teeth image cut out from the rendering image Rb, thereby generating a frame image Mb reflecting the simulation result. The data generating device 10 converts the portion of the frame image C showing the upper and lower teeth image c into a portion corresponding to the upper and lower teeth image cut out from the rendering image Rc, thereby generating a frame image Mc reflecting the simulation result. The data generating device 10 converts the portion of the frame image D showing the upper and lower teeth image d into a portion corresponding to the upper and lower teeth image cut out from the rendering image Rd, thereby generating a frame image Md reflecting the simulation result. The data generating device 10 can generate simulation video data in which the subject reflects the simulation result and makes a smile by associating these multiple frame images Ma, Mb, Mc, and Md in chronological order.

[0074] 3, the data generating device 10 generates a simulation result of a smile based on the simulation video data stored in the storage device 12, and outputs the generated simulation result (S12). For example, the data generating device 10 generates a simulation video in which post-simulation frame images Ma, Mb, Mc, and Md as shown in Fig. 13 are switched in time series based on the simulation video data, and displays the generated simulation video on the display 40 and transmits the data of the simulation video from the communication device 18 to an external device.

[0075] As described above, the data generating device 10 according to the first embodiment can generate simulation video data showing the face of the subject reflecting the results of the simulation of dental treatment, using video data obtained by photographing the subject with some of the upper and lower dentitions exposed, and three-dimensional data of the upper and lower dentitions of the subject. This allows the user to use the camera 20 and the three-dimensional scanner 30 to convert the video of the subject obtained by photographing the subject before dental treatment into a video assuming that the subject has undergone dental treatment, and thus allows the user to easily and accurately simulate the face of the subject after dental treatment.

[0076] The data generating device 10 first uses rendering images with the viewpoint tilted at a rough angle in the horizontal or vertical direction to roughly search for rendering images including upper and lower dental row images that match in appearance with the upper and lower dental row images included in each frame image by pattern matching, and then generates rendering images with the viewpoint tilted at a fine angle in the horizontal or vertical direction based on the rendering images generated (extracted) by rough pattern matching, and searches in detail for rendering images including upper and lower dental row images that match in appearance with the upper and lower dental row images included in each frame image by pattern matching. For example, the data generating device 10 tries pattern matching between the upper and lower teeth images of the frame image using rendering images in which the viewpoint is changed at a rough angle (e.g., 3 degree intervals) in both the horizontal and vertical directions, and then tries pattern matching between the upper and lower teeth images of the frame image using rendering images in which the viewpoint is changed at a fine angle (e.g., 1 degree intervals) in both the horizontal and vertical directions, using a rendering image with a relatively high matching level (above a first predetermined value) as a reference from among the tried pattern matchings, and then considers pattern matching to be achieved when a rendering image with a very high matching level (above a second predetermined value that is higher than the first predetermined value) is obtained from among the tried pattern matchings. This allows the data generating device 10 to reduce the processing load for pattern matching and shorten the processing time for pattern matching, compared to performing detailed pattern matching from the beginning.

[0077] The video data acquired by the camera 20 may be input to the data generating device 10 in real time while the subject is being photographed. In this case, the data generating device 10 may execute the data generation process of Fig. 3 following the input of video data via the video data interface 13 while the subject is being photographed, thereby generating simulation video data in real time while the subject is being photographed. As a result, the simulation video data is generated simultaneously with or immediately after the subject is photographed by the camera 20, so that the user can immediately show the subject the simulation results, improving user convenience.

[0078] The data generating device 10 according to the above-mentioned embodiment 1 is configured to first perform a simulation of dental treatment on the three-dimensional data acquired by the three-dimensional scanner 30 in S6, and then, in S7 and S8, pattern match an image of the upper and lower teeth extracted from one frame image selected from the multiple frame images constituting the video data with a rendering image reflecting the simulation, but may also be configured as follows.

[0079] For example, the data generating device 10 may generate multiple rendering images based on the 3D data acquired by the 3D scanner 30 before the simulation is reflected, without executing the process of S6, and may perform pattern matching between the upper and lower dentition image extracted from one frame image selected from the multiple frame images constituting the video data and the rendering image before the simulation in S7 and S8. The data generating device 10 may then perform a simulation of dental treatment for the upper and lower dentition for each of the multiple rendering images corresponding to the multiple frame images extracted in S7 and S8. In this way, the data generating device 10 can generate simulation video data in which the subject smiles with the simulation results reflected.

[0080] The data generating device 10 may receive the color temperature of the light source used when the video was captured by the camera 20. When generating a rendering image in S9, the data generating device 10 may adjust the color of the simulated upper and lower teeth image included in the rendering image so as to approximate the color of the upper and lower teeth of the subject at the time of capture based on the color temperature of the acquired light source. This allows the data generating device 10 to generate video data using the simulated upper and lower teeth image with a color similar to the color of the upper and lower teeth of the subject. Furthermore, the data generating device 10 may receive parameters related to the light source used when the video was captured by the camera 20, such as the position or amount of light of the light source, refraction or reflection of light, and may generate a rendering image based on these parameters. The data generating device 10 may use AI (artificial intelligence) that has been machine-learned to generate an optimal rendering image according to the condition of light or shadow detectable from each frame image included in the video data, the color of the upper and lower teeth, and the like, thereby shortening the processing time for adjusting the light, shadow, or color in the rendering image.

[0081] <Embodiment 2> A data generating device 10 according to a second embodiment of the present disclosure will be described in detail with reference to Fig. 14. Note that, in the data generating device 10 according to the second embodiment, only the parts different from the data generating device 10 according to the first embodiment will be described, and the parts that are the same as those in the data generating device 10 according to the first embodiment will be denoted by the same reference numerals and will not be described repeatedly.

[0082] FIG. 14 is a flowchart for explaining an example of data generation processing executed by the data generating device 10 according to the second embodiment. In the data generation processing according to the second embodiment shown in FIG. 14, an example will be explained in which a simulation image is first generated by pattern matching using only a first frame image included in the video data, and then a simulation image corresponding to at least one other second frame image included in the video data is generated using the simulation image corresponding to the first frame image. In the second embodiment, the position (viewpoint) of the camera 20 and the position of the subject are fixed while the subject is being captured as a video. That is, in each of the frame images A to D acquired by the camera 20, the position (viewpoint) of the camera 20 and the position of the upper and lower teeth that are the subject are the same.

[0083] As shown in FIG. 14, similar to the processes of S1 to S5 shown in FIG. 3, the data generating device 10 extracts a plurality of frame images (for example, frame images A, B, C, and D in FIG. 4) that constitute the video data (S21), performs preprocessing (S22 to S24), and extracts upper and lower dental row images corresponding to the upper and lower dental row from each frame image (S25).

[0084] Similar to the processing of S6 shown in FIG. 3, the data generating device 10 performs a simulation of dental treatment on the upper and lower teeth (S26), and then generates a plurality of rendering images including two-dimensional upper and lower teeth viewed from multiple directions based on the three-dimensional data of the simulated upper and lower teeth after dental treatment, while roughly pattern matching the upper and lower teeth image included in the first frame image extracted in S25 with the upper and lower teeth image included in each of the plurality of rendering images (S27), and further performs detailed pattern matching between the upper and lower teeth image included in the first frame image extracted in S25 with the upper and lower teeth image included in each of the plurality of rendering images (S28).

[0085] For example, in S27, the data generating device 10 roughly searches for a rendering image including an upper and lower dental row image that matches in appearance with the upper and lower dental row image included in frame image A (first frame image) by pattern matching. Then, in S28, the data generating device 10 generates a rendering image with a viewpoint tilted at a small angle in the horizontal or vertical direction based on the rendering image generated (extracted) in S27, and searches in detail for a rendering image including an upper and lower dental row image that matches in appearance with the upper and lower dental row image included in frame image A (first frame image) by pattern matching.

[0086] The data generating device 10 generates a first rendering image corresponding to the upper and lower teeth image included in the first frame image based on the viewpoint of the camera 20 in the horizontal and vertical directions of the upper and lower teeth image included in the first frame image extracted by pattern matching in S27 and S28, and also generates a second rendering image corresponding to the upper and lower teeth image included in another second frame image using the generated first rendering image (S29).

[0087] For example, in S29, the data generating device 10 generates a rendering image Ra (first rendering image) corresponding to the upper and lower teeth image included in frame image A (first frame image), and uses the generated rendering image Ra (first rendering image) to generate rendering images Rb to Rd (second rendering images) corresponding to the upper and lower teeth images included in each of frame images B to D (second frame images).

[0088] The data generating device 10 cuts out the portions corresponding to the upper and lower tooth row images extracted from each frame image in S25 from each of the generated rendering images Ra, Rb, Rc, and Rd (S30).

[0089] The data generating device 10 converts the portion of each frame image showing the upper and lower dentition image into a portion corresponding to the upper and lower dentition image cut out from the rendering image in S30, and processes the portion where the converted upper and lower dentition portion joins with the unconverted portion other than the upper and lower dentition smoothly, thereby generating simulation video data showing the subject's face reflecting the simulation results, and stores the data in the storage device 12 (S31).

[0090] The data generating device 10 generates a simulation result of a smile based on the simulation moving image data stored in the storage device 12, and outputs the generated simulation result (S32).

[0091] As described above, data generating device 10 according to the second embodiment first generates a simulation image corresponding to a first frame image included in moving image data by pattern matching using only the first frame image, and then generates a simulation image corresponding to a second frame image included in the moving image data using the simulation image corresponding to the first frame image, so that there is only one frame image for which pattern matching is performed. This allows data generating device 10 to reduce the processing load for pattern matching and shorten the processing time for pattern matching compared to performing pattern matching on each of the multiple frame images that make up the moving image data.

[0092] <Embodiment 3> A data generating device 10 according to a third embodiment of the present disclosure will be described in detail with reference to Fig. 15 to Fig. 19. Note that, in the data generating device 10 according to the third embodiment, only the parts different from the data generating device 10 according to the first embodiment will be described, and the parts that are the same as those in the data generating device 10 according to the first embodiment will be denoted by the same reference numerals and description thereof will not be repeated.

[0093] Fig. 15 is a flowchart for explaining an example of data generation processing executed by the data generating device 10 according to embodiment 3. The data generation processing according to embodiment 3 shown in Fig. 15 explains an example in which AI is used to estimate the area (shape) of the upper and lower teeth rows in each frame image constituting video data, and further AI is used to estimate each rendering image corresponding to the upper and lower teeth rows included in each frame image.

[0094] As shown in FIG. 15, similar to the processes S1 to S4 shown in FIG. 3, the data generating device 10 extracts a plurality of frame images (for example, frame images A, B, C, and D in FIG. 4) that constitute video data (S41) and performs preprocessing (S42 to S44).

[0095] The data generating device 10 estimates the upper and lower teeth rows in each frame image constituting the video data using AI (S45). Specifically, the data generating device 10 estimates the area (shape) of the upper and lower teeth rows in each frame image based on input data including each frame image, using an estimation model 130 for estimating the area (shape) of the upper and lower teeth rows in each frame image based on the input data.

[0096] The estimation model 130 includes a known neural network used in image recognition processing using deep learning, such as a convolutional neural network (CNN), a generative adversarial network (GAN), a recurrent neural network (RNN), or a long short-term memory network (LSTM), as well as parameters related to the neural network.

[0097] Here, supervised learning for the estimation model 130 will be described with reference to Fig. 16. Fig. 16 is a diagram for explaining an overview of supervised learning executed by the data generating device 10 according to the third embodiment. As shown in Fig. 16, in the learning phase, the data generating device 10 executes the learning program 110 to machine-learn the estimation model 130 based on the learning data 120 including an input 1 and an input 2 (correct answer). In the utilization phase, the data generating device 10 uses the trained estimation model 130 to estimate an output based on the input 1.

[0098] FIG. 17 is a diagram for explaining the input and output of supervised learning in the data generating device 10 according to the third embodiment. As shown in FIG. 17, a frame image including an image of upper and lower teeth is used as the input 1, which is the problem data of the machine learning. Data for identifying the region of the upper and lower teeth in the frame image of the input 1 is used as the input 2, which is the correct answer data of the machine learning. For example, the data for identifying the region of the upper and lower teeth in the frame image includes data for distinguishing the region of the upper and lower teeth from the region of the part other than the upper and lower teeth in the frame image. For example, a frame image in which the data "1" is assigned to the region of the upper and lower teeth and the data "0" is assigned to the region of the part other than the upper and lower teeth is used for the input 2. Furthermore, data for identifying the region of the upper tooth row and the lower tooth row may be distinguished, and for example, a frame image in which the data "1" is assigned to the region of the upper tooth row, "2" is assigned to the region of the lower tooth row, and "0" is assigned to the region of the part other than the upper and lower tooth row may be used for the input 2. As the output that the data generating device 10 estimates based on the input 1, the estimation result of the data that identifies the regions of the upper and lower dentition in the frame image of the input 1 is used.

[0099] That is, in the learning phase, the data generating device 10 estimates data (output) that specifies the region of the upper and lower teeth in the frame image based on a frame image (input 1) including the upper and lower teeth image, compares the estimated data (output) that specifies the region of the upper and lower teeth with data (input 2) that specifies the region of the upper and lower teeth in the frame image that is the correct answer, and optimizes the estimation model 130 by not updating the parameters included in the estimation model 130 if the two match, and updating the parameters included in the estimation model 130 so that the two match if they do not match. The trained estimation model 130 thus trained by machine learning is stored in the storage device 12 as the data generating program 100. In the utilization phase, the data generating device 10 uses the trained estimation model 130 to estimate the upper and lower teeth image (output) included in the frame image, i.e., the shape of the upper and lower teeth shown by the frame image, based on the frame image (input 1) including the upper and lower teeth image.

[0100] Returning to Fig. 14, the data generating device 10 performs a simulation of dental treatment for the upper and lower teeth (S46) in the same manner as the process of S6 shown in Fig. 3, and then estimates each rendering image (for example, rendering images Ra, Rb, Rc, and Rd in Fig. 11) corresponding to each upper and lower teeth image (for example, upper and lower teeth images a, b, c, and d in Fig. 5) estimated in S45 based on each frame image A, B, C, and D constituting the video data using AI (S47). Specifically, the data generating device 10 estimates the rendering image corresponding to the upper and lower teeth image based on the input data including the upper and lower teeth image included in each frame image estimated in S45 and a plurality of rendering images including the upper and lower teeth image, using an estimation model 130 for estimating the rendering image corresponding to the upper and lower teeth image included in each frame image.

[0101] FIG. 18 is a diagram for explaining the input and output of supervised learning in the data generating device 10 according to the third embodiment. As shown in FIG. 18, upper and lower teeth images extracted based on each frame image and three-dimensional data of the upper and lower teeth are used as input 1, which is problem data for machine learning. A rendering image corresponding to the upper and lower teeth images included in each frame image of input 1 is used as input 2, which is correct answer data for machine learning. An estimation result of the rendering image corresponding to the upper and lower teeth images included in each frame image of input 1 is used as the output estimated by the data generating device 10 based on input 1.

[0102] That is, in the learning phase, the data generating device 10 estimates a rendering image (output) corresponding to the upper and lower teeth image included in each frame image based on the upper and lower teeth image and the three-dimensional data of the upper and lower teeth (input 1) extracted based on each frame image, compares each estimated rendering image (output) with a rendering image (input 2) corresponding to the upper and lower teeth image included in each frame image as a correct answer, and optimizes the estimation model 130 by not updating the parameters included in the estimation model 130 if the two match, and updating the parameters included in the estimation model 130 so that the two match if they do not match. The trained estimation model 130 thus trained is stored in the storage device 12 as the data generating program 100. In the utilization phase, the data generating device 10 uses the trained estimation model 130 to estimate a rendering image (output) corresponding to the upper and lower teeth image included in each frame image based on the upper and lower teeth image and the three-dimensional data of the upper and lower teeth (input 1) extracted based on each frame image.

[0103] Returning to FIG. 15, the data generating device 10 cuts out the portions corresponding to the upper and lower tooth row images extracted from each frame image in S47 from each of the generated rendering images Ra, Rb, Rc, and Rd (S48).

[0104] The data generating device 10 converts the portion of each frame image showing the upper and lower dentition image into a portion corresponding to the upper and lower dentition image cut out from the rendering image in S48, and processes the converted upper and lower dentition portion to smoothly connect the fusion portion with the unconverted portion other than the upper and lower dentition, thereby generating simulation video data showing the subject's face reflecting the simulation results, and stores the data in the storage device 12 (S49).

[0105] The data generating device 10 generates a simulation result of a smile based on the simulation moving image data stored in the storage device 12, and outputs the generated simulation result (S50).

[0106] As described above, the data generating device 10 according to the third embodiment uses AI to estimate the shape of the upper and lower teeth rows from each frame image constituting the video data, and therefore can extract the upper and lower teeth row images from each frame image constituting the video data with greater accuracy. Furthermore, the data generating device 10 uses AI to estimate each rendering image corresponding to the upper and lower teeth rows included in each frame image, and therefore can extract each rendering image corresponding to the upper and lower teeth rows included in each frame image with greater accuracy. This allows the data generating device 10 to generate simulation video data with greater accuracy that shows the subject smiling, reflecting the simulation results.

[0107] <Fourth embodiment> A data generating device 10 according to a fourth embodiment of the present disclosure will be described in detail with reference to Fig. 19. Note that, in the data generating device 10 according to the fourth embodiment, only the parts different from the data generating device 10 according to the first embodiment will be described, and the parts that are the same as those in the data generating device 10 according to the first embodiment will be denoted by the same reference numerals and will not be described repeatedly.

[0108] FIG. 19 is a flowchart for explaining an example of data generation processing executed by the data generating device 10 according to the fourth embodiment. In the fourth embodiment shown in FIG. 19, the data generating system 1 includes a CT (Computed Tomography) imaging device (not shown). The CT imaging device is an X-ray imaging device that rotates a transmitter and a receiver of X-rays, which are a type of radiation, around the face of the subject to perform computed tomography of the upper and lower jaws of the subject. By imaging the upper and lower jaws of the subject using the CT imaging device, the user can obtain three-dimensional volume (voxel) data of hard tissue parts (bones, teeth, etc.) other than soft tissue parts (skin, gums, etc.) around the upper and lower jaws of the subject. The user can generate a tomographic image or an external image of the subject by performing tomographic processing on the volume data of the subject obtained by the CT imaging device. In the following, the three-dimensional surface coordinate data acquired by the three-dimensional scanner 30 is also referred to as "IOS data", and the three-dimensional volume data acquired by the CT imaging device is also referred to as "CT data". In addition, since both the IOS data and the CT data correspond to three-dimensional data of the upper and lower dentition, the IOS data and the CT data can be collectively referred to as "three-dimensional data."

[0109] In the data generation process according to the fourth embodiment, in addition to IOS data of the subject's upper and lower dentition obtained by the three-dimensional scanner 30, CT data of the subject's upper and lower jaws including the upper and lower dentition obtained by a CT imaging device is input.

[0110] 19, the data generating device 10 generates surface shape data indicating the shapes of the surfaces of hard tissue portions (bones, teeth, etc.) in the upper and lower jaws of a subject based on CT data input from a CT imaging device (S61). The data generating device 10 separates the surface shape data generated in S61 for the upper and lower jaws (S62).

[0111] The data generating device 10 generates multiple rendering images including two-dimensional upper and lower teeth viewed from multiple directions based on the IOS data of the upper and lower teeth of the subject acquired by the three-dimensional scanner 30 and the surface shape data of the upper and lower jaws generated in S62. Since the IOS data includes position information of each point group indicating the surface of the upper and lower teeth, it can accurately represent the position of each tooth and the bite of the upper and lower teeth. On the other hand, the surface shape data of the upper and lower jaws generated based on the CT data can accurately represent the surface shape of the upper and lower teeth and the tooth roots of each tooth. Therefore, the data generating device 10 can generate rendering images that more accurately represent the shape and position of the two-dimensional upper and lower teeth by combining the IOS data of the upper and lower teeth and the surface shape data of the upper and lower jaws.

[0112] The data generating device 10 generates a plurality of rendering images including two-dimensional upper and lower teeth rows viewed from multiple directions based on the rendering images using the IOS data and CT data as described above, and roughly pattern matches the upper and lower teeth row images included in each frame image extracted in S5 with the upper and lower teeth row images included in each of the plurality of rendering images (S63), and further performs detailed pattern matching between the upper and lower teeth row images included in each frame image extracted in S5 with the upper and lower teeth row images included in each of the plurality of rendering images (S64).

[0113] Thereafter, the data generation device 10 can execute the processes of S9 to S12 in a manner similar to the data generation process of embodiment 1 shown in FIG. 3, thereby generating simulation video data showing the subject's face reflecting the results of the dental treatment simulation.

[0114] As described above, the data generating device 10 of embodiment 4 can generate rendering images that more accurately represent the two-dimensional shape and position of the upper and lower dental arches, not only by using IOS data acquired by the three-dimensional scanner 30, but also by using CT data acquired by a CT imaging device, and can accurately generate simulation video data in which the subject is smiling, reflecting the simulation results, based on the video data of the subject acquired by the camera 20 and the rendering image.

[0115] In addition, in the data generating device 10 according to either of the second and third embodiments, the IOS data acquired by the three-dimensional scanner 30 and the CT data acquired by the CT imaging device can be adopted as the three-dimensional data input to the data generating device 10. In this case, in either of the data generating devices 10 according to the second and third embodiments, the process of S61 to S64 of the data generating process according to the fourth embodiment shown in FIG. 19 is executed to execute pattern matching. In addition, the three-dimensional data input to the data generating device 10 is not limited to the IOS data obtained by directly scanning the upper and lower teeth of the subject using the optical method of the three-dimensional scanner 30 as described above, and the CT data obtained by performing computed tomography of the upper and lower jaws of the subject using a CT imaging device, but may also include surface shape data obtained by scanning a dental jaw model created from an impression of the upper and lower teeth of the subject using a desktop scanner.

[0116] The data generating device 10 is configured to separate the surface shape data for the upper and lower jaws, and generate a rendering image corresponding to the upper jaw and a rendering image corresponding to the lower jaw based on the surface shape data of the upper jaw and the surface shape data of the lower jaw, respectively, but the rendering images of the upper and lower jaws may be generated based on the surface shape data of the upper and lower jaws combined. The data generating device 10 may then perform pattern matching between the upper and lower dentition images using the generated rendering images of the upper and lower jaws. Furthermore, since the biting motion video data also captures the occlusal state, the data generating device 10 may additionally use IOS data of the occlusal state when generating the rendering images of the upper and lower jaws.

[0117] <Embodiment 5> A data generating device 10 according to a fifth embodiment of the present disclosure will be described in detail with reference to Fig. 20. Note that, in the data generating device 10 according to the fifth embodiment, only the parts different from the data generating device 10 according to the first embodiment will be described, and the parts that are the same as those in the data generating device 10 according to the first embodiment will be denoted by the same reference numerals and will not be described repeatedly.

[0118] FIG. 20 is a flowchart for explaining an example of a data generation process executed by the data generating device 10 according to the fifth embodiment. In the fifth embodiment shown in FIG. 20, the data generating system 1 includes a CT imaging device (not shown) as in the data generating device 10 according to the fourth embodiment. Furthermore, in the fifth embodiment, the data generating system 1 includes a facial data measuring device (not shown). By using the facial data measuring device, a user can obtain three-dimensional data indicating the shape of the subject's face surface. In the following, the three-dimensional volume data acquired by the CT imaging device is also referred to as "CT data", and the three-dimensional data indicating the shape of the face surface acquired by the facial data measuring device is also referred to as "FS data". In addition, the CT data and the FS data can be collectively referred to as "three-dimensional data".

[0119] In the data generation process of embodiment 5, in addition to CT data of the subject's upper and lower jaws, including the upper and lower dentition, acquired by a CT imaging device, FS data indicating the shape of the subject's facial surface acquired by a facial data measuring device is input.

[0120] 20, the data generating device 10 generates composite data by synthesizing the CT data input from the CT imaging device and the FS data input from the facial data measuring device (S71). The composite data generated by the data generating device 10 can represent the surface shape of the hard tissue parts (bones, teeth, etc.) in the upper and lower jaw of the subject that can be identified by the CT data, and the surface shape of the soft tissue parts (skin, etc.) of the face including the upper and lower jaw of the subject that can be identified by the FS data.

[0121] The data generating device 10 performs a simulation of dental treatment on the jawbone and upper and lower dentition based on the CT data of the composite data (S72). The data generating device 10 changes the FS data of the composite data in accordance with the dental treatment on the jawbone and upper and lower dentition using the CT data (S73). That is, the data generating device 10 performs a simulation of dental treatment on the jawbone and upper and lower dentition by changing the CT data, and deforms soft tissue parts (skin, gums, etc.) by changing the FS data in accordance with the change in the surface shape of the hard tissue parts (bones, teeth, etc.) in the upper and lower jaw due to the change in the CT data.

[0122] The data generating device 10 generates multiple rendering images of the oral cavity including two-dimensional upper and lower teeth rows viewed from multiple directions based on the composite data including the CT data and FS data after reflecting the simulation of dental treatment. While generating multiple rendering images including two-dimensional upper and lower teeth rows viewed from multiple directions based on the rendering images using the CT data and FS data as described above, the data generating device 10 roughly matches the upper and lower teeth row images included in each frame image extracted in S5 with the upper and lower teeth row images included in each of the multiple rendering images (S74), and furthermore, performs detailed pattern matching between the upper and lower teeth row images included in each frame image extracted in S5 with the upper and lower teeth row images included in each of the multiple rendering images (S75).

[0123] Then, the data generating device 10 generates rendering images corresponding to the upper and lower dental row images contained in each frame image based on the viewpoint of the camera 20 in the horizontal and vertical directions of the upper and lower dental row images contained in each frame image extracted by pattern matching in S74 and S75 (S76).

[0124] The data generating device 10 cuts out an image corresponding to the oral cavity including the upper and lower dentition extracted from each frame image in S5 from each of the generated rendering images Ra, Rb, Rc, Rd (S77). Since each of the rendering images Ra, Rb, Rc, Rd is an image generated based on the CT data and the FS data, the simulation image cut out from each of the rendering images Ra, Rb, Rc, Rd includes the soft tissue portion after the upper and lower dentition and the jaw of the subject are deformed.

[0125] The data generating device 10 converts the image of the facial part that is the target of conversion in each frame image into an image of the facial part including the upper and lower dentition cut out from the rendering image in S77, and processes it so as to smoothly connect the fusion parts of the converted facial part and other facial parts that have not been converted, thereby generating simulation video data showing the subject's face reflecting the simulation results, and stores it in the storage device 12 (S78).

[0126] The data generating device 10 generates a simulation result of a smile based on the simulation moving image data stored in the storage device 12, and outputs the generated simulation result (S79).

[0127] As described above, the data generating device 10 according to the fifth embodiment can generate a rendering image that more accurately represents the shape and position of the oral cavity including the two-dimensional upper and lower dentition, using not only the CT data acquired by the CT imaging device but also the FS data acquired by the facial data measuring device, and can extract a simulation image including the soft tissue part after the upper and lower dentition and the jawbone of the subject are deformed from the rendering image. As a result, the data generating device 10 can accurately generate simulation video data that reflects the simulation result of deforming the jawbone in addition to the upper and lower dentition, based on the video data of the subject acquired by the camera 20 and the rendering image, such that the subject smiles.

[0128] In the data generating device 10 according to either of the second and third embodiments, the CT data acquired by the CT imaging device and the FS data acquired by the facial data measuring device can be adopted as the three-dimensional data input to the data generating device 10. In this case, in the data generating device 10 according to either of the second and third embodiments, the processing of S71 to S76 of the data generating process according to the fifth embodiment shown in FIG. 20 is executed to generate a rendering image.

[0129] <Sixth embodiment> A data generating device 10 according to a sixth embodiment of the present disclosure will be described in detail with reference to Fig. 21. Note that, in the data generating device 10 according to the sixth embodiment, only the parts different from the data generating device 10 according to the first embodiment will be described, and the parts that are the same as those in the data generating device 10 according to the first embodiment will be denoted by the same reference numerals and will not be described repeatedly.

[0130] In data generating device 10 according to the sixth embodiment, camera 20 is configured to capture a moving image of a subject (e.g., a face of a subject) from a plurality of viewpoints. For example, FIG. 21 is a diagram showing an application example of data generating system 1 and data generating device 10 according to the sixth embodiment. As shown in FIG. 21, data generating system 1 according to the sixth embodiment includes camera 20A that captures a face of a subject from substantially in front of the subject (e.g., in front of the subject's face) and camera 20B that captures a face of the subject from substantially to the side of the subject (e.g., from the side of the subject's face).

[0131] The data generating device 10 may execute pattern matching (for example, the processes of S7 and S8 in FIG. 3) using upper and lower tooth row images extracted from each frame image constituting the video data acquired by the camera 20A and upper and lower tooth row images extracted from each frame image constituting the video data acquired by the camera 20B. At this time, the data generating device 10 may preferentially use, for pattern matching, the video data acquired by the camera 20A or the video data image acquired by the camera 20B, whichever has higher pattern matching accuracy.

[0132] For example, the camera 20A captures the subject's face from the front, and the horizontal shapes of the upper and lower teeth extracted from each frame image are likely to be accurately represented. Therefore, the data generating device 10 may perform the horizontal pattern matching of S8 by preferentially using the upper and lower teeth images extracted from each frame image constituting the video data captured by the camera 20A.

[0133] In addition, the camera 20B captures the subject's face from the side, and the vertical shapes of the upper and lower teeth extracted from each frame image are likely to be accurately represented. Therefore, the data generating device 10 may perform pattern matching in the vertical direction by preferentially using the upper and lower teeth images extracted from each frame image constituting the video data acquired by the camera 20B.

[0134] As described above, the data generating device 10 according to the sixth embodiment can use video data obtained by shooting a subject from multiple viewpoints to accurately generate simulation video data in which the subject smiles, reflecting the simulation results.

[0135] The camera is not limited to being disposed approximately in front of the subject or approximately to the side of the subject, but may be disposed approximately diagonally in front of the subject. That is, the data generation system 1 may include one camera disposed at any one of approximately in front of the subject, approximately diagonally in front of the subject, and approximately to the side of the subject, or may include multiple cameras disposed at any two or more of approximately in front of the subject, approximately diagonally in front of the subject, and approximately to the side of the subject. The data generation device 10 may obtain video data from multiple cameras that simultaneously capture the subject from multiple viewpoints, and may perform pattern matching (for example, the processes of S7 and S8 in FIG. 3) using the video data obtained from the multiple cameras.

[0136] <Embodiment 7> A data generating device 10 according to a seventh embodiment of the present disclosure will be described in detail with reference to Fig. 22 and Fig. 23. Note that, in the data generating device 10 according to the seventh embodiment, only the parts different from the data generating device 10 according to the third embodiment will be described, and the parts that are the same as those in the data generating device 10 according to the third embodiment will be denoted by the same reference numerals and will not be described repeatedly.

[0137] The data generating device 10 of embodiment 7, when converting the portion of each frame image showing the upper and lower dentition image into a portion corresponding to the upper and lower dentition image cut out from the rendering image, uses AI to smoothly connect the fusion area between the converted upper and lower dentition portion and the unconverted portion other than the upper and lower dentition in a more natural manner, and applies brightness and shading to the dentition according to the brightness and shading of the face.

[0138] Fig. 22 is a diagram for explaining input and output of supervised learning in the data generating device according to embodiment 7. As shown in Fig. 22, a frame image in which the upper and lower teeth images are masked, a rendering image corresponding to the upper and lower teeth images, and an image for identifying the positions of the upper and lower teeth are used as input 1, which is problem data for machine learning.

[0139] For example, Fig. 23 is a diagram showing an example of input data for supervised learning in the data generating device according to the seventh embodiment. As shown in Fig. 23(A), a frame image in which an upper and lower tooth row image is masked is generated based on a frame image included in the video data, and is used as input 1. As shown in Fig. 23(B), a rendering image in which only a portion of the frame image corresponding to the upper and lower tooth row image is extracted is generated based on three-dimensional data of the upper and lower tooth row reflecting the simulation result, and is used as input 1. As shown in Fig. 23(C), an image for identifying the position of the upper and lower tooth row image in the frame image is generated based on a frame image included in the video data, and is used as input 1.

[0140] As shown in Fig. 22, a frame image including an image of upper and lower teeth included in video data is used as input 2, which is the correct answer data for machine learning. That is, a frame image showing upper and lower teeth in a natural state before simulation is used as input 2. As an output estimated by the data generating device 10 based on input 1, an estimated result of a frame image in which the image of upper and lower teeth after simulation is converted into an image having natural shading and coloring like the image of upper and lower teeth before simulation is used.

[0141] In the learning phase, the data generating device 10 estimates the estimation result of the frame image in which the upper and lower teeth images after the simulation are converted into the upper and lower teeth images before the simulation based on the frame image in which the upper and lower teeth images shown in FIG. 23(A) are masked, the rendering image corresponding to the upper and lower teeth images shown in FIG. 23(B), and the image for identifying the position of the upper and lower teeth shown in FIG. 23(C) (input 1), that is, the frame image (output) in which the upper and lower teeth images shown in FIG. 23(B) are synthesized in a natural state in the masked portion of the image shown in FIG. 23(A), and compares the estimated frame image (output) with a frame image (input 2) in which the upper and lower teeth are projected in a natural state before the simulation, and if the two match, the parameters included in the estimation model 130 are not updated, while if they do not match, the parameters included in the estimation model 130 are updated so that the two match, thereby optimizing the estimation model 130. The learned estimation model 130 thus machine-learned is stored in the storage device 12 as the data generating program 100. In the application phase, the data generating device 10 estimates a frame image in which upper and lower teeth images are synthesized in a natural state based on a frame image including upper and lower teeth images included in the video data and a rendering image of the upper and lower teeth reflecting the simulation result. In the frame image including the upper and lower teeth after the simulation estimated in this way, the fusion part between the converted upper and lower teeth part and the unconverted part other than the upper and lower teeth is smoothly connected, and the brightness and shading of the teeth are added according to the brightness and shading of the face, and the brightness of the upper and lower teeth part reflecting the simulation result is also adjusted so that the unevenness of the surface of the upper and lower teeth is expressed in a natural state. Note that the above-mentioned match determination between the frame image (output) and the frame image (input 2) may be a match determination of the degree of naturalness of the images.

[0142] <Modification> The present disclosure is not limited to the above-described embodiment, and various modifications and applications are possible. Modifications that can be applied to the present disclosure will be described below.

[0143] In the data generating device 10, the camera 20 may be a three-dimensional camera capable of detecting the three-dimensional position coordinates of the subject. For example, the camera 20 may be a ToF (Time of Flight) depth camera capable of detecting the three-dimensional position coordinates of the subject by measuring the distance to the subject using reflection of light (for example, infrared rays), or a stereo camera capable of detecting the three-dimensional position coordinates of the subject by using multiple (for example, two) cameras to capture the same space from different positions. That is, the video data input to the data generating device 10 includes three-dimensional data that is the position information (coordinates of each axis in the vertical, horizontal, and height directions) of each of the point clouds (multiple points) indicating the surface of the upper and lower teeth of the subject to be captured.

[0144] The data generating device 10 may be a cloud-type server device. That is, the calculation device 11 (calculation unit) may have a function as a cloud-type server device. For example, a user or a subject may take a picture of the subject's face using a user terminal such as a smartphone, and upload and register video data obtained by taking the picture of the subject and three-dimensional data of the subject's upper and lower teeth rows to the cloud-type data generating device 10. The data generating device 10 may convert the upper and lower teeth row images included in the frame images of the video data into simulated upper and lower teeth row images cut out from the rendering image to generate a simulation video and distribute it to the user terminal, so that the user or subject can view or save the simulation video using the user terminal.

[0145] When generating a rendering image based on three-dimensional data reflecting the simulation results, the data generating device 10 may generate the rendering image by performing optical simulation using multiple light sources based on the depth of the upper and lower teeth corresponding to the unevenness of the upper and lower teeth surfaces. In this way, the data generating device 10 can adjust the brightness of the upper and lower teeth shown in the rendering image to match the unevenness of the upper and lower teeth surfaces, so that the upper and lower teeth after the simulation can be represented in the rendering image in a more natural way.

[0146] 13, the simulation moving image data is composed of a plurality of frame images Ma, Mb, Mc, and Md. Therefore, the data generating device 10 may generate a plurality of frame images Ma, Mb, Mc, and Md showing the face of the subject reflecting the simulation result based on the simulation moving image data, and display them on the display 40.

[0147] The data generating system 1 and the data generating device 10 according to each of the above-mentioned first to seventh embodiments may have each other's configurations and functions alone or in combination.

[0148] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is indicated by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. Note that the configurations exemplified in the present embodiment and the configurations exemplified in the modified examples can be appropriately combined. [Explanation of symbols]

[0149] 1 Data generation system, 10 Data generation device, 11 Computing device, 12 Storage device, 13 Video data interface, 14 Three-dimensional data interface, 15 Display interface, 16 Peripheral device interface, 17 Media reading device, 18 Communication device, 20, 20A, 20B Camera, 30 Three-dimensional scanner, 40 Display, 51 Keyboard, 52 Mouse, 60 Removable disk, 100 Data generation program, 110 Learning program, 120 Learning data, 130 Estimation model.

Claims

1. A data generating device that generates simulation video data showing a face of a subject reflecting a simulation result of dental treatment, an input unit to which video data obtained by photographing the subject in a state where a part of the upper and lower dentitions is exposed and three-dimensional data of the upper and lower dentitions of the subject are input; a calculation unit that generates the simulation video data showing the face of the subject reflecting the simulation result of the dental treatment based on the part of the rows of teeth included in each of a plurality of frame images constituting the video data and three-dimensional data of the upper and lower rows of teeth, The calculation unit is generating at least one rendering image including two-dimensional upper and lower teeth corresponding to the part of the teeth included in each of the plurality of frame images based on three-dimensional data of the upper and lower teeth of the subject in which a simulation of the dental treatment is performed on the upper and lower teeth of the subject and the simulation result is reflected; A data generating device that generates the simulation video data by converting an image of the portion of the teeth before the simulation results included in each of the plurality of frame images are reflected into an image of the portion of the teeth of the upper and lower teeth reflecting the simulation results included in at least one rendering image.

2. The data generating device according to claim 1 , wherein the calculation unit generates the simulation moving image data in response to the moving image data being input to the input unit while the subject is being photographed.

3. The data generating device of claim 1, wherein the calculation unit uses an estimation model based on input data including the at least one rendering image and the multiple frame images to convert an image of the portion of the teeth before the simulation results included in each of the multiple frame images are reflected into an image of the portion of the teeth of the upper and lower teeth reflecting the simulation results included in the at least one rendering image.

4. The data generating device of claim 1, wherein when converting an image of the portion of the teeth before the simulation results included in each of the plurality of frame images are reflected into an image of the portion of the upper and lower teeth reflecting the simulation results included in the at least one rendering image, the calculation unit deforms the jawbone of the subject in the at least one rendering image to match the upper and lower teeth, and smoothly connects a fusion portion between the converted portion and the non-converted portion in the simulation video data.

5. The data generating device according to claim 1 , wherein the calculation unit generates the at least one rendering image by performing an optical simulation using a plurality of light sources based on the depths of the upper and lower dental arches.

6. The data generating device according to claim 1 , wherein the calculation unit generates a plurality of frame images showing the face of the subject reflecting the simulation result based on the simulation video data.

7. The data generating device according to claim 1 , wherein the video data includes video data obtained by capturing video using at least one of a monocular camera, a stereo camera, and a depth camera.

8. The data generating device of claim 1, wherein the calculation unit estimates the area of ​​the portion of the dentition in the frame image based on input data including frame images constituting the video data, using an estimation model for estimating the area of ​​the portion of the dentition in the frame image based on the input data.

9. The data generating device of claim 1, wherein the calculation unit estimates a rendering image corresponding to the portion of the teeth row included in the frame images constituting the video data based on input data including the shape of the portion of the teeth row included in the frame images constituting the video data and three-dimensional data of the upper and lower teeth row, using an estimation model for estimating a rendering image corresponding to the portion of the teeth row included in the frame images.

10. The data generating device of claim 1, wherein the three-dimensional data input to the input unit includes at least one of three-dimensional data including position information of each of the point clouds representing the surfaces of the upper and lower teeth, three-dimensional data obtained by computed tomography of the upper and lower teeth, and three-dimensional data representing the shape of the surface of the subject's face.

11. The input unit receives a color temperature of a light source used when the subject is photographed, The data generating device according to claim 1 , wherein the calculation unit generates the simulation video data based on the color temperature.

12. The data generating device according to claim 1 , wherein the calculation unit has a function as a cloud-type server device.

13. A data generation method for generating simulation video data showing a face of a subject reflecting a result of a computer-based dental treatment simulation, comprising: inputting video data obtained by photographing the subject with some of the upper and lower dentitions exposed, and three-dimensional data of the upper and lower dentitions of the subject; generating the simulation video data showing the face of the subject reflecting the simulation result of the dental treatment based on the part of the dentition included in each of a plurality of frame images constituting the video data and three-dimensional data of the upper and lower dentitions, The generating step includes: generating at least one rendering image including two-dimensional upper and lower teeth corresponding to the part of the teeth included in each of the plurality of frame images based on three-dimensional data of the upper and lower teeth of the subject, the three-dimensional data being obtained by performing a simulation of the dental treatment on the upper and lower teeth of the subject and reflecting the simulation result; and generating the simulation video data by converting an image of the portion of the teeth before the simulation results included in each of the plurality of frame images are reflected into an image of the portion of the teeth of the upper and lower teeth reflecting the simulation results included in the at least one rendering image.

14. A data generation program for generating simulation video data showing a face of a subject reflecting a result of a dental treatment simulation, comprising: On the computer, inputting video data obtained by photographing the subject with some of the upper and lower dentitions exposed, and three-dimensional data of the upper and lower dentitions of the subject; generating the simulation video data showing the face of the subject reflecting the simulation result of the dental treatment based on the part of the rows of teeth included in each of a plurality of frame images constituting the video data and three-dimensional data of the upper and lower rows of teeth; The generating step includes: generating at least one rendering image including two-dimensional upper and lower teeth corresponding to the part of the teeth included in each of the plurality of frame images based on three-dimensional data of the upper and lower teeth of the subject, the three-dimensional data being obtained by performing a simulation of the dental treatment on the upper and lower teeth of the subject and reflecting the simulation result; and generating the simulation video data by converting an image of the portion of the teeth before the simulation results included in each of the plurality of frame images are reflected into an image of the portion of the teeth of the upper and lower teeth reflecting the simulation results included in at least one rendering image.

Images