Data generation device, data generation method, and data generation program

Abstract

To provide a technique which allows a user to easily and accurately confirm the jaw movement.SOLUTION: A data generation device 10 comprises: a moving image data interface 13 and a three-dimensional data interface 14 which receive inputs of moving image data obtained by imaging an object person who is performing the jaw movement in such a state that a partial tooth row in upper and lower tooth rows is exposed and three-dimensional data of the upper and lower tooth rows of the object person; and a calculation device 11 which generates jaw movement-related data related to the jaw movement of the object person on the basis of the partial tooth row included in each of a plurality of frame images constituting the moving image data and the three-dimensional data of the upper and lower tooth rows.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a data generation device, a data generation method, and a data generation program that generate jaw movement-related data related to the jaw movement of a subject.

Background Art

[0002] Conventionally, for patients with symptoms such as pain in the temporomandibular joint or patients who complain of malocclusion, jaw movement may be observed to confirm occlusion. Also, even during orthodontic treatment or the creation of dentures or prostheses, it is necessary to observe the patient's jaw movement to confirm occlusion. Confirmation of jaw movement can be performed using a plaster model that reproduces the shape of the patient's dentition and an articulator. Also, a jig for measuring the positional relationship between the upper and lower jaws and the movement data of the lower jaw can be attached to the patient, and the patient's jaw movement can be confirmed based on the operation of the jig. Furthermore, in recent years, attempts have been made to simulate the patient's jaw movement on a computer by combining three-dimensional data of the dentition obtained by a CT (Computed Tomography) device that performs computerized tomography of the patient's dentition using X-rays or a three-dimensional scanner that optically scans the patient's dentition, and three-dimensional data corresponding to the operation of an articulator or jig worn by the patient.

[0003] Also, with the progress of image processing technology by a computer, it is possible to create a virtual three-dimensional surface model of the surface of the patient's face, and the patient's surface model can be used to confirm changes in the shape of the face or jaw movement due to orthodontic treatment. For example, Patent Document 1 discloses a dental computed tomography device capable of creating a virtual three-dimensional surface model of the patient's face.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As described above, conventionally, in checking the jaw movement of a patient, it is necessary to have the patient wear a bite appliance or a jig, so the preparation is troublesome. Furthermore, since the patient has to perform jaw movement while wearing the bite appliance or jig, an unnecessary load is imposed on the patient's jaw movement, making it difficult to accurately check the jaw movement. Also, in the dental computed tomography apparatus disclosed in Patent Document 1, it is necessary to provide an imaging station including a plurality of members such as a color camera, a laser device, and an illumination device, so the preparation is troublesome. For this reason, there is a need for a technique that can easily and accurately check the jaw movement of a patient.

[0006] The present disclosure has been made to solve such problems, and an object thereof is to provide a technique that can easily and accurately check jaw movement.

Means for Solving the Problems

[0007] According to an example of the present disclosure, there is provided a data generation device that generates jaw movement-related data related to the jaw movement of a subject. The data generation device includes an input unit to which video data obtained by photographing a subject who is performing jaw movement with a part of the upper and lower dental arches exposed and three-dimensional data of the upper and lower dental arches of the subject are input, and a part included in each of a plurality of frame images constituting the video data. Vertical and horizontal A calculation unit that generates jaw movement-related data related to the jaw movement of the subject based on the dental arch and the three-dimensional data of the upper and lower dental arches. Using one camera, at the front of the subject or from a viewpoint tilted by a predetermined angle in the horizontal or vertical direction from the front Vertical and horizontal Based on the dental arch and the three-dimensional data of the upper and lower dental arches, By performing pattern matching with the upper and lower dental arches included in the rendering image generated in this way, and searching for the three-dimensional position of the three-dimensional data of the upper and lower dental arches during jaw movement it is provided with an arithmetic unit that generates jaw movement-related data related to the jaw movement of the subject. The calculation unit generates a plurality of rendering images at a viewpoint tilted by a predetermined angle in the horizontal or vertical direction from the front of the subject based on the three-dimensional data of the upper and lower dental arches.

[0008] According to an example of the present disclosure, there is provided a data generation method for generating jaw movement-related data related to the jaw movement of a subject by a computer. The data generation method includes a part of the upper and lower dental arches. Vertical and horizontal A subject who is performing jaw movement with a part of the dental arch exposed​Using one camera, at the front of the subject or from a viewpoint tilted by a predetermined angle in the horizontal or vertical direction from the front A step of inputting video data obtained by photographing and three-dimensional data of the upper and lower dental arches of a subject, and a part included in each of a plurality of frame images constituting the video data Vertical and horizontal Based on the dental arch and the three-dimensional data of the upper and lower dental arches By performing pattern matching with the upper and lower dental arches included in the rendering image generated in this way, and searching for the three-dimensional position of the three-dimensional data of the upper and lower dental arches during jaw movement And a step of generating jaw movement-related data related to the jaw movement of the subject. The generating step includes the step of generating a plurality of rendering images at a viewpoint tilted by a predetermined angle in the horizontal or vertical direction from the front of the subject based on the three-dimensional data of the upper and lower dental arches.

[0009] According to an example of the present disclosure, a data generation program for generating jaw movement-related data related to the jaw movement of a subject is provided. The data generation program causes a computer to perform a step of inputting video data obtained by photographing a subject who is making a jaw movement with a part of the upper and lower dental arches exposed, and three-dimensional data of the upper and lower dental arches of the subject, and a step of generating jaw movement-related data related to the jaw movement of the subject based on a part of the dental arch included in each of a plurality of frame images constituting the video data. Vertical and horizontal A subject who is making a jaw movement with a part of the dental arch exposed Using one camera, at the front of the subject or from a viewpoint tilted by a predetermined angle in the horizontal or vertical direction from the front A step of inputting video data obtained by photographing and three-dimensional data of the upper and lower dental arches of the subject, and a part included in each of a plurality of frame images constituting the video data Vertical and horizontal Based on the dental arch and the three-dimensional data of the upper and lower dental arches By performing pattern matching with the upper and lower dental arches included in the rendering image generated in this way, and searching for the three-dimensional position of the three-dimensional data of the upper and lower dental arches during jaw movement And a step of generating jaw movement-related data related to the jaw movement of the subject. The generating step includes the step of generating a plurality of rendering images at a viewpoint tilted by a predetermined angle in the horizontal or vertical direction from the front of the subject based on the three-dimensional data of the upper and lower dental arches.

Advantages of the Invention

[0010] According to the present disclosure, a user can obtain jaw movement-related data related to the jaw movement of a subject by using video data obtained by photographing a subject who is making a jaw movement with a part of the upper and lower dental arches exposed and three-dimensional data of the upper and lower dental arches of the subject, so that the jaw movement of the subject can be easily and accurately confirmed.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0012] <Embodiment 1> Embodiment 1 of the present disclosure will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and the description thereof will not be repeated.

[0013] [Application Example] With reference to FIG. 1, an application example of the data generation system 1 and the data generation device 10 according to Embodiment 1 will be described. FIG. 1 is a diagram showing an application example of the data generation system 1 and the data generation device 10 according to Embodiment 1.

[0014] For example, in order to check the occlusion for patients with symptoms such as temporomandibular joint pain or patients who complain of malocclusion, the jaw movement of the patient may be observed, or in order to check the occlusion during orthodontic treatment of teeth, fabrication of dentures or prostheses, the jaw movement of the patient may be observed. Conventionally, in checking the jaw movement of a subject such as a patient, it has been troublesome to prepare because it is necessary to have the subject wear an occlusal appliance or a jig. Therefore, the data generation system 1 and the data generation device 10 according to Embodiment 1 are configured to be able to easily and accurately check the jaw movement of the subject by using video data obtained by photographing a subject who is making a jaw movement with a part of the upper and lower dental arches exposed, and three-dimensional data of the upper and lower dental arches of the subject.

[0015] Specifically, as shown in FIG. 1, the data generation system 1 includes a camera 20 capable of video shooting, a three-dimensional scanner 30 capable of acquiring three-dimensional data of an object, and a data generation device 10.

[0016] The camera 20 may be any camera that can shoot a subject in video (for example, an RGB camera), such as a digital camera or a video camera. In Embodiment 1, for example, a monocular camera is exemplified as the camera 20. The video data acquired by the camera 20 is composed of a plurality of frame images arranged in time series. A frame image is a single still image showing a two-dimensional object to be photographed as seen from the viewpoint of the camera 20 directed at the object to be photographed (for example, the face of the subject). For example, when the frame rate of the camera 20 is set to 24 fps, the video data acquired by the camera 20 includes 24 frame images per second. When it is desired to check the jaw movement with higher time resolution, as the camera 20, for example, a camera with a high frame rate such as 120 fps or 240 fps may be used.

[0017] The three-dimensional scanner 30 is a so-called intraoral scanner (IOS: Intra Oral Scanner) that can optically image the inside of a subject's oral cavity by means of confocal method or triangulation method. Specifically, the three-dimensional scanner 30 obtains, as three-dimensional data, the position information (coordinates of each axis in the vertical, horizontal, and height directions) of each point in a point cloud (a plurality of points) indicating the surface of the object to be imaged (object) inside the oral cavity by scanning the inside of the oral cavity using an optical sensor or the like. That is, the three-dimensional data is position data including the position information of each position of the point cloud constituting the surface of an object (for example, the upper and lower dental arches inside the oral cavity) that is the scanning target placed on a certain coordinate space. When the upper and lower dental arches inside the oral cavity are scanned by the three-dimensional scanner 30, three-dimensional data including the position information of each point in the point cloud indicating the surface of the upper and lower dental arches is obtained. With such three-dimensional data, it is possible to specify the shape of the object (upper and lower dental arches) placed on the coordinate space. Hereinafter, a collection of three-dimensional data capable of specifying the shape of an object (upper and lower dental arches) placed on the coordinate space is also referred to as "shape data". Also, data capable of specifying the position of an object (upper and lower dental arches) placed on the coordinate space is also referred to as "position data".

[0018] The user uses the camera 20 to shoot a video of a subject whose jaw is moving with a part of the upper and lower dental arches exposed. Further, the user can obtain three-dimensional data of the upper and lower dental arches inside the subject's oral cavity by scanning the inside of the subject's oral cavity using the three-dimensional scanner 30. The video data obtained by the camera 20 and the three-dimensional data obtained by the three-dimensional scanner 30 are input into the data generation device 10. Note that the data generation device 10 is not limited to the case where video data obtained by photographing a subject is input, and for example, a plurality of still images obtained by continuously photographing a subject in time series may be input.

[0019] "User" includes operators (such as doctors) or assistants (such as dental assistants, dental technicians, nurses, etc.) in various fields such as dentistry, oral surgery, orthopedic surgery, plastic surgery, and cosmetic surgery. "Subject" includes patients in dentistry, oral surgery, orthopedic surgery, plastic surgery, and cosmetic surgery. "Upper and lower dental arches" includes both the upper dental arch and the lower dental arch in the oral cavity, or either the upper dental arch or the lower dental arch. That is, "upper and lower dental arches" includes at least one of the upper dental arch and the lower dental arch. "A partial dental arch of the upper and lower dental arches" includes both a partial dental arch included in the upper dental arch and a partial dental arch included in the lower dental arch, or either a partial dental arch included in the upper dental arch or a partial dental arch included in the lower dental arch. That is, "a partial dental arch of the upper and lower dental arches" includes at least one of a partial dental arch included in the upper dental arch and a partial dental arch included in the lower dental arch.

[0020] Based on the video data of the subject acquired by the camera 20 and the three-dimensional data of the upper and lower dental arches of the subject acquired by the three-dimensional scanner 30, the data generation device 10 generates jaw movement-related data (jaw movement-related data Ra, Rb, Rc, Rd shown in FIG. 1) related to the jaw movement of the subject.

[0021] Thereby, the user can obtain jaw movement-related data of the subject by using the video data obtained by photographing the subject whose jaw is moving with a partial dental arch of the upper and lower dental arches exposed and the three-dimensional data of the upper and lower dental arches of the subject, so that the jaw movement of the subject can be easily and accurately confirmed.

[0022] [Hardware Configuration of Identification Device] With reference to FIG. 2, the hardware configuration of the data generation device 10 according to Embodiment 1 will be described. FIG. 2 is a block diagram showing the hardware configuration of the data generation device 10 according to Embodiment 1. The data generation device 10 may be realized by, for example, a general-purpose computer or a computer dedicated to the data generation system 1.

[0023] As shown in FIG. 2, the data generation device 10 includes, as main hardware elements, an arithmetic unit 11, a storage device 12, a video data interface 13, a three-dimensional data interface 14, a display interface 15, a peripheral device interface 16, a media reader 17, and a communication device 18.

[0024] The arithmetic unit 11 is an arithmetic entity (computer) that executes various processes by executing various programs, and is an example of an "arithmetic unit". The arithmetic unit 11 is composed of, for example, a processor such as a CPU (central processing unit) or an MPU (Micro-processing unit). Note that a processor, which is an example of the arithmetic unit 11, has a function of executing various processes by executing a program, but some or all of these functions may be implemented using a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). The "processor" is not limited to a narrow sense processor that executes processing in a stored program manner such as a CPU or an MPU, and may include a hardwired circuit such as an ASIC or an FPGA. Therefore, the "processor", which is an example of the arithmetic unit 11, can also be read as a processing circuitry whose processing is defined in advance by computer-readable code and / or a hardwired circuit. Note that the arithmetic unit 11 may be composed of one chip or a plurality of chips. Furthermore, some functions of the arithmetic unit 11 may be provided in a server device (for example, a cloud-type server device) not shown in the figure.

[0025] The memory device 12 includes a volatile memory area (e.g., a working area) that temporarily stores program codes, work memories, etc. when the arithmetic unit 11 executes an arbitrary program. For example, the memory device 12 includes a volatile memory such as a DRAM (dynamic random access memory) and an SRAM (static random access memory), or a non-volatile memory such as a ROM (Read Only Memory) and a flash memory. Further, the memory device 12 may be an SSD (solid state drive) or an HDD (hard disk drive), etc. In this case, the arithmetic unit 11 may include a volatile memory such as a DRAM and an SRAM, or a non-volatile memory such as a ROM and a flash memory.

[0026] In addition, in the first embodiment, an example in which the volatile memory area and the non-volatile memory area are included in the same memory device 12 is shown, but the volatile memory area and the non-volatile memory area may be included in different memory parts. For example, the arithmetic unit 11 may include a volatile memory area, and the memory device 12 may include a non-volatile memory area. The data generation device 10 may include a microcomputer including the arithmetic unit 11 and the memory device 12.

[0027] The memory device 12 stores the data generation program 100. The data generation program 100 describes the content of the data generation process for generating the subject's jaw movement-related data based on the video data of the subject whose jaw is moving and the three-dimensional data of the subject's upper and lower dental arches.

[0028] The video data interface 13 is an interface for connecting the camera 20 and is an example of an "input unit". The video data interface 13 realizes the input / output of data (e.g., video data) between the data generation device 10 and the camera 20. The data generation device 10 and the camera 20 are connected via wired or wireless (WiFi, Bluetooth (registered trademark), etc.) using a cable.

[0029] The three-dimensional data interface 14 is an interface for connecting the three-dimensional scanner 30 and is an example of an "input unit". The three-dimensional data interface 14 realizes the input and output of data (for example, three-dimensional data) between the data generation device 10 and the three-dimensional scanner 30. The data generation device 10 and the three-dimensional scanner 30 are connected via wired or wireless (such as WiFi, Bluetooth (registered trademark), etc.) using a cable.

[0030] The display interface 15 is an interface for connecting the display 40. The display interface 15 realizes the input and output of data between the data generation device 10 and the display 40.

[0031] The peripheral device interface 16 is an interface for connecting peripheral devices such as the keyboard 51 and the mouse 52. The peripheral device interface 16 realizes the input and output of data between the data generation device 10 and the peripheral devices.

[0032] The media reader 17 reads various data stored in the removable disk 60, which is a storage medium, or writes various data to the removable disk 60. For example, the media reader 17 may acquire the data generation program 100 from the removable disk 60, or may write the jaw movement-related data of the subject generated by the arithmetic unit 11 to the removable disk 60.

[0033] The communication device 18 transmits and receives data to and from an external device via wired or wireless communication. For example, the communication device 18 may transmit the jaw movement-related data of the subject generated by the arithmetic unit 11 to an external device.

[0034] [Data Generation Process] Referring to FIGS. 3 to 11, the data generation process executed by the data generation device 10 according to Embodiment 1 will be described. In the data generation process according to Embodiment 1 shown in FIGS. 3 to 11, an example of extracting jaw movement-related data corresponding to each frame image by pattern matching using each frame image included in the video data will be described. FIG. 3 is a flowchart for explaining an example of the data generation process executed by the data generation device 10 according to Embodiment 1. Each STEP (hereinafter, referred to as "S") shown in FIG. 3 is realized by the arithmetic device 11 of the data generation device 10 executing the data generation program 100.

[0035] As shown in FIG. 3, after the video data of the subject with jaw movement acquired by the camera 20 is input into the data generation device 10, a plurality of frame images constituting the video data are extracted from the video data (S1).

[0036] For example, FIG. 4 is a diagram for explaining the processing content of STEP1 in the data generation process. As shown in FIG. 4, when the camera 20 captures a video of a subject with jaw movement in a state where a part of the dental arch is exposed, the video data acquired by the camera 20 includes a plurality of frame images A, B, C, D showing the subject with jaw movement in a state where a part of the dental arch is exposed. For example, the frame image A includes a part of the dental arch with the upper and lower dental arches closed, and the frame images B to D include a part of the dental arch with the upper and lower dental arches open. The frame images A to D show how the upper and lower dental arches change from the closed state to the open state. Note that while the subject is being video-recorded, the position (viewpoint) of the camera 20 and the position of the subject being photographed are not necessarily fixed. Therefore, the appearance of the upper and lower dental arches as seen from the camera position shown by each frame image A to D acquired in time series may be different from each other. The data generation device 10 extracts a plurality of frame images showing the state where the subject gradually opens the upper and lower dental arches from the closed state from the video data acquired by the camera 20.

[0037] Returning to Fig. 3, in S2 to S4, the data generation device 10 executes preprocessing for generating the jaw movement related data of the subject.

[0038] Specifically, when distortion corresponding to the lens of the camera 20 occurs in at least one frame image among a plurality of frame images, the data generation device 10 corrects the distortion in the at least one frame image (S2). By performing such distortion correction, the data generation device 10 can improve the accuracy of pattern matching between the shape of the upper and lower dental arches included in the frame image and the shape of the upper and lower dental arches embodied based on the three-dimensional data acquired by the three-dimensional scanner 30, which will be described later.

[0039] The data generation device 10 detects the positions of each of the imaging target (for example, the face of the subject) and the camera 20 (S3). By using the positions of each of the imaging target and the camera 20 for pattern matching described later, the data generation device 10 can improve the accuracy of pattern matching, further reduce the processing load imposed on pattern matching, and shorten the processing time of pattern matching. Note that the data generation device 10 may acquire and store the positions of each of the imaging target and the camera 20 at the start or during imaging by the camera 20 in the storage device 12. In the process of S3, the data generation device 10 may search for and acquire the positions of each of the imaging target and the camera 20 stored by the camera 20, or may search for and acquire the positions of each of the imaging target and the camera 20 stored by the storage device 12.

[0040] The data generation device 10 detects the distance between the imaging target (for example, the face of the target person) and the camera 20 (S4). By using the distance between the imaging target and the camera 20 for pattern matching described later, the data generation device 10 can reduce the processing load imposed on the pattern matching and shorten the processing time of the pattern matching. Note that the data generation device 10 may acquire the distance between the imaging target and the camera 20 at the start or during imaging by the camera 20 and store it in the storage device 12. In the process of S3, the data generation device 10 may search for and acquire the distance stored by the camera 20, or may search for and acquire the distance stored by the storage device 12.

[0041] Note that the data generation device 10 may perform pattern matching described later based on at least one of the positional relationship between the imaging target and the camera 20 and the distance between the imaging target and the camera 20 acquired in S3.

[0042] The data generation device 10 extracts upper and lower dental arch images corresponding to the upper and lower dental arches from each frame image extracted in S1 (S5).

[0043] For example, FIG. 5 is a diagram for explaining the processing content of STEP5 in the data generation process. As shown in FIG. 5, the data generation device 10 extracts upper and lower dental arch images corresponding to the upper and lower dental arches from each of the frame images A, B, C, and D. For example, the data generation device 10 compares a template image showing the shape of the upper and lower dental arches stored in advance with each of the frame images A, B, C, and D, and extracts the upper and lower dental arch images from each of the frame images A, B, C, and D by pattern matching. Note that the data generation device 10 is not limited to pattern matching that compares the shapes of the upper and lower dental arches, and may extract the upper and lower dental arch images from each of the frame images A, B, C, and D by pattern matching that compares the edge shapes of the upper and lower dental arches. The "upper and lower dental arch image" is an image including both the upper dental arch and the lower dental arch in the oral cavity, or either the upper dental arch or the lower dental arch. Further, the "upper and lower dental arch image" is an image including a partial dental arch of the upper and lower dental arches.

[0044] The data generation device 10 extracts an upper and lower dental arch image a showing the upper and lower dental arches of the subject from the face portion of the subject included in the frame image A. The data generation device 10 extracts an upper and lower dental arch image b showing the upper and lower dental arches of the subject from the face portion of the subject included in the frame image B. The data generation device 10 extracts an upper and lower dental arch image c showing the upper and lower dental arches of the subject from the face portion of the subject included in the frame image C. The data generation device 10 extracts an upper and lower dental arch image d showing the upper and lower dental arches of the subject from the face portion of the subject included in the frame image D.

[0045] Returning to FIG. 3, after the three-dimensional data of the upper and lower dental arches of the subject acquired by the three-dimensional scanner 30 is input, the data generation device 10 generates a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions based on the three-dimensional data. A "rendering image" is an image generated by performing processing or editing on certain data, and in Embodiment 1, it is a two-dimensional image generated by performing processing or editing on the three-dimensional data acquired by the three-dimensional scanner 30.

[0046] As described above, the three-dimensional data acquired by the three-dimensional scanner 30 includes the position information of each point in the point cloud showing the surface of the upper and lower dental arches of the subject. The data generation device 10 can generate a rendering image showing a two-dimensional upper and lower dental arch viewed from a certain viewpoint by imaging the point cloud showing the surface of the upper and lower dental arches using the three-dimensional data acquired by the three-dimensional scanner 30. Further, the data generation device 10 can generate a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions by changing the viewpoint in multiple directions.

[0047] Based on the three-dimensional data of the upper and lower dental arches acquired by the three-dimensional scanner 30, while generating a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions, the data generation device 10 roughly pattern-matches the upper and lower dental arch images included in each frame image extracted in S5 with the upper and lower dental arch images included in each of the plurality of rendering images (S6), and further, the upper and lower dental arch images included in each frame image extracted in S5 are precisely pattern-matched with the upper and lower dental arch images included in each of the plurality of rendering images (S7).

[0048] For example, FIGS. 6 to 10 are diagrams for explaining the processing contents of STEP6 and STEP7 in the data generation process. As shown in FIG. 6, the data generation device 10 virtually views an object (upper and lower dental arches) placed on a coordinate space that can be specified based on the three-dimensional data while changing the viewpoint (the position of the camera 20) and the angle of view, thereby generating a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions in the horizontal or vertical direction. Then, the data generation device 10 searches for a rendering image of the upper and lower dental arch image that looks the same as the upper and lower dental arch image included in each frame image by pattern matching, and generates (extracts) a rendering image corresponding to the upper and lower dental arch image included in each frame image. Note that "search" means searching for a rendering image that matches the upper and lower dental arch image included in the frame image from among a plurality of rendering images by trying pattern matching between each of the plurality of rendering images and the upper and lower dental arch image included in the frame image once or multiple times. When the data generation device 10 generates a rendering image corresponding to the upper and lower dental arch image included in the frame image, it stores the position data of the object (upper and lower dental arches) when the rendering image was generated. Alternatively, the data generation device 10 stores the change amount (the movement amount of the object) of the position data of the object (upper and lower dental arches) when each rendering image corresponding to each frame image was generated.

[0049] With reference to FIGS. 7 and 8, an example of generating a rendering image will be described. As shown in FIG. 7, it is assumed that the data generation device 10 gradually moves the viewpoint when generating a rendering image horizontally from a state where the viewpoint is directed at the front of the subject's face (upper and lower dental arches), and generates each rendering image including the two-dimensional upper and lower dental arches viewed from each viewpoint in the horizontal direction.

[0050] Specifically, the data generation device 10 generates a rendering image TC including the two-dimensional upper and lower dental arches when the subject's face (upper and lower dental arches) is viewed from the front, based on the three-dimensional data acquired by the three-dimensional scanner 30. The data generation device 10 generates a rendering image TL including the two-dimensional upper and lower dental arches when the viewpoint is slightly tilted to the left in the horizontal direction starting from the front of the subject's face (upper and lower dental arches) (for example, when the viewpoint is tilted 10 degrees to the left in the horizontal direction starting from the front). The data generation device 10 generates a rendering image TR including the two-dimensional upper and lower dental arches when the viewpoint is slightly tilted to the right in the horizontal direction starting from the front of the subject's face (upper and lower dental arches) (for example, when the viewpoint is tilted 10 degrees to the right in the horizontal direction starting from the front).

[0051] As shown in FIG. 8, it is assumed that the data generation device 10 gradually moves the viewpoint vertically from a state where the viewpoint is directed at the front of the subject's face (upper and lower dental arches) when generating a rendering image, and generates each rendering image including the two-dimensional upper and lower dental arches viewed from each viewpoint in the vertical direction.

[0052] Specifically, based on the three-dimensional data acquired by the three-dimensional scanner 30, the data generation device 10 generates a rendering image TC including the two-dimensional upper and lower dental arches when viewing the face (upper and lower dental arches) of the subject from the front. The data generation device 10 generates a rendering image TD including the two-dimensional upper and lower dental arches when the viewpoint is tilted slightly downward in the vertical direction starting from the front of the face (upper and lower dental arches) of the subject (for example, when the viewpoint is tilted downward by 10 degrees in the vertical direction starting from the front). The data generation device 10 generates a rendering image TU including the two-dimensional upper and lower dental arches when the viewpoint is tilted slightly upward in the vertical direction starting from the front of the face (upper and lower dental arches) of the subject (for example, when the viewpoint is tilted upward by 10 degrees in the vertical direction starting from the front).

[0053] Although not shown, based on the rendering image TL, the data generation device 10 generates a rendering image including the two-dimensional upper and lower dental arches when the viewpoint is tilted slightly downward in the vertical direction (for example, when the viewpoint is tilted downward by 10 degrees in the vertical direction starting from the front), and a rendering image including the two-dimensional upper and lower dental arches when the viewpoint is tilted slightly upward in the vertical direction (for example, when the viewpoint is tilted upward by 10 degrees in the vertical direction starting from the front). Based on the rendering image TR, the data generation device 10 generates a rendering image including the two-dimensional upper and lower dental arches when the viewpoint is tilted slightly downward in the vertical direction (for example, when the viewpoint is tilted downward by 10 degrees in the vertical direction starting from the front), and a rendering image including the two-dimensional upper and lower dental arches when the viewpoint is tilted slightly upward in the vertical direction (for example, when the viewpoint is tilted upward by 10 degrees in the vertical direction starting from the front).

[0054] In this way, the data generation device 10 can generate a plurality of rendering images with the viewpoints tilted step by step in the horizontal direction and a plurality of rendering images with the viewpoints tilted step by step in the vertical direction. In the data generation device 10, an angle at which the front of the face (upper and lower dental arches) of the subject is tilted in the horizontal direction or the vertical direction is preset. The finer the angle is set, the more rendering images the data generation device 10 can generate based on viewpoints from multiple directions.

[0055] In step S6, the data generation device 10 roughly searches by pattern matching for a rendering image including an upper and lower dental arch image that looks the same as the upper and lower dental arch image included in each frame image, using rendering images with the viewpoints tilted at a rough angle (for example, at 3-degree intervals) in the horizontal direction or the vertical direction. Then, based on the rendering image generated (extracted) in S6, in step S7, the data generation device 10 generates rendering images with the viewpoints tilted at a fine angle (for example, at 1-degree intervals) in the horizontal direction or the vertical direction, and searches in detail by pattern matching for a rendering image including an upper and lower dental arch image that looks the same as the upper and lower dental arch image included in each frame image.

[0056] With reference to FIGS. 9 and 10, an example of pattern matching will be described. As shown in FIG. 9, the data generation device 10 performs pattern matching in the horizontal direction between the upper and lower dental arch images extracted in S5 based on each frame image constituting the video data and each rendering image including a two-dimensional upper and lower dental arch viewed from each viewpoint in the horizontal direction, to extract the viewpoint of the camera 20 in the horizontal direction of the upper and lower dental arch images extracted from each frame image.

[0057] For example, the data generation device 10 performs pattern matching in the horizontal direction between the upper and lower dental arch images a extracted from the frame image A showing the state where the subject's upper and lower dental arches are closed, and a plurality of rendering images TC, TL, TR including the two-dimensional upper and lower dental arches viewed from each viewpoint in the horizontal direction generated based on the three-dimensional data acquired by the three-dimensional scanner 30, and extracts the rendering image corresponding to the upper and lower dental arch image a from among the plurality of rendering images TC, TL, TR.

[0058] Specifically, the data generation device 10 performs pattern matching in the horizontal direction between the shape (appearance) of the upper and lower dental arches included in the upper and lower dental arch image a and the shape (appearance) of the upper and lower dental arches included in each of the plurality of rendering images TC, TL, TR, and extracts, from among the plurality of rendering images TC, TL, TR, the rendering image including the shape of the upper and lower dental arches that is most approximated in the horizontal direction to the shape of the upper and lower dental arches included in the upper and lower dental arch image a. At this time, the data generation device 10 may perform pattern matching in the horizontal direction based on the characteristics of the teeth. For example, the data generation device 10 may perform pattern matching in the horizontal direction for the shape near the cutting edge of the incisors, near the occlusal surface of the canines or molars, etc. In the example of FIG. 9, the data generation device 10 extracts the rendering image TC as the rendering image that matches the upper and lower dental arch image a in the horizontal direction. Thereby, the data generation device 10 can extract the viewpoint of the camera 20 in the horizontal direction of the upper and lower dental arch image a. The data generation device 10 can also extract the viewpoint of the camera 20 in the horizontal direction of the upper and lower dental arch images included in each of the frame images B to D in the same manner as the frame image A for the other frame images B to D.

[0059] As shown in FIG. 10, the data generation device 10 performs pattern matching in the vertical direction between the upper and lower dental arch images extracted in S5 based on each frame image constituting the moving image data and each rendering image including the two-dimensional upper and lower dental arches viewed from each viewpoint in the vertical direction, and extracts the viewpoint of the camera 20 in the vertical direction of the upper and lower dental arch images extracted from each frame image.

[0060] For example, the data generation device 10 pattern-matches in the vertical direction the upper and lower dental arch image a extracted from the frame image A showing the state where the subject's upper and lower dental arches are closed, and a plurality of rendering images TC, TD, TU including the two-dimensional upper and lower dental arches viewed from each viewpoint in the vertical direction generated based on the three-dimensional data acquired by the three-dimensional scanner 30, and extracts from the plurality of rendering images TC, TD, TU the rendering image corresponding to the upper and lower dental arch image a.

[0061] Specifically, the data generation device 10 pattern-matches in the vertical direction the shape (appearance) of the upper and lower dental arches included in the upper and lower dental arch image a and the shape (appearance) of the upper and lower dental arches included in each of the plurality of rendering images TC, TD, TU, and thereby extracts from the plurality of rendering images TC, TD, TU the rendering image including the shape of the upper and lower dental arches that is most approximated in the vertical direction to the shape of the upper and lower dental arches included in the upper and lower dental arch image a. At this time, the data generation device 10 may perform pattern matching in the vertical direction based on the characteristics of the teeth. For example, the data generation device 10 may perform pattern matching in the vertical direction for the shape near the incisal edge of the incisors, near the occlusal surface of the canines or molars, etc. In the example of FIG. 10, the data generation device 10 extracts the rendering image TC as the rendering image that matches the upper and lower dental arch image a in the vertical direction. Thereby, the data generation device 10 can extract the viewpoint of the camera 20 in the vertical direction of the upper and lower dental arch image a. The data generation device 10 can also extract the viewpoints of the camera 20 in the vertical direction of the upper and lower dental arch images included in each of the frame images B to D in the same manner as the frame image A for the other frame images B to D.

[0062] Note that the data generation device 10 is not limited to generating a plurality of rendering images based on viewpoints in multiple directions while performing pattern matching, and may generate a plurality of rendering images based on viewpoints in multiple directions in advance before performing pattern matching.

[0063] When the data generation device 10 performs pattern matching between the upper and lower dental arch images extracted from each frame image and the plurality of rendering images in S6 and S7, it may utilize the positions of each of the imaging target (for example, the face of the subject) and the camera 20 acquired in S3. For example, based on the positions of each of the imaging target (for example, the face of the subject) and the camera 20, the data generation device 10 may enlarge or reduce the upper and lower dental arch images included in the plurality of rendering images according to the size of the upper and lower dental arch images. Thereby, the data generation device 10 can improve the accuracy of pattern matching by image recognition, further reduce the processing burden involved in pattern matching, and shorten the processing time of pattern matching.

[0064] Furthermore, since the data generation device 10 corrects the distortion in the frame image in advance in S2, the upper and lower dental arch images extracted from each frame image that is the target of pattern matching can be made clearer. Thereby, the data generation device 10 can improve the accuracy of pattern matching by image recognition.

[0065] Returning to FIG. 3, based on the viewpoints of the camera 20 in the horizontal and vertical directions of the upper and lower dental arch images included in each frame image extracted by the pattern matching in S6 and S7, the data generation device 10 generates each rendering image corresponding to the upper and lower dental arch images included in each frame image (S8).

[0066] For example, FIG. 11 is a diagram for explaining the processing content of STEP8 in the data generation process. As shown in FIG. 11, based on the viewpoints in the horizontal and vertical directions in each of the frame images A to D extracted in S6 and S7, the data generation device 10 generates each rendering image Ra to Rd viewed from each of the viewpoints.

[0067] Specifically, the data generation device 10 generates a rendering image Ra in which the upper and lower dental arches are closed in accordance with the upper and lower dental arch images a included in the frame image A. The data generation device 10 generates a rendering image Rb in which the upper and lower dental arches in the closed state indicated by the rendering image Ra are changed to a slightly opened state in accordance with the upper and lower dental arch images b included in the frame image B. The data generation device 10 generates a rendering image Rc in which the upper and lower dental arches in the closed state indicated by the rendering image Ra are changed to a more opened state than the rendering image Rb in accordance with the upper and lower dental arch images c included in the frame image C. The data generation device 10 generates a rendering image Rd in which the upper and lower dental arches in the closed state indicated by the rendering image Ra are changed to a more opened state than the rendering image Rc in accordance with the upper and lower dental arch images d included in the frame image D.

[0068] Returning to FIG. 3, the data generation device 10 generates jaw movement-related data related to the jaw movement of the subject indicated by each of the frame images A, B, C, D based on the generated rendering images Ra, Rb, Rc, Rd, and stores the jaw movement-related data in the storage device 12 in association with the data of each of the frame images A, B, C, D (S9).

[0069] For example, the data generation device 10 stores, as jaw movement-related data, shape data that can identify the shape of each of the upper and lower dental arches, i.e., the upper dental arch and the lower dental arch, which are the scanning targets of the three-dimensional scanner 30. Further, the data generation device 10 stores position data (spatial coordinates) that can identify the positions of each of the upper and lower dental arches based on the respective rendering images Ra to Rd of the upper and lower dental arches corresponding to the upper and lower dental arch images included in each of the frame images A to D. Alternatively, the data generation device 10 stores, as jaw movement-related data, the change between the positions of each of the upper and lower dental arches that can be identified based on the rendering image Ra and the positions of each of the upper and lower dental arches that can be identified based on the rendering image Rb, stores the change between the positions of each of the upper and lower dental arches that can be identified based on the rendering image Rb and the positions of each of the upper and lower dental arches that can be identified based on the rendering image Rc, and stores the change between the positions of each of the upper and lower dental arches that can be identified based on the rendering image Rc and the positions of each of the upper and lower dental arches that can be identified based on the rendering image Rd.

[0070] Based on the jaw movement-related data corresponding to each of the rendering images Ra, Rb, Rc, and Rd indicating the jaw movement of the subject stored in the storage device 12, the data generation device 10 generates a simulation result of the jaw movement and outputs the generated simulation result (S10). For example, the data generation device 10 generates a simulation video that switches the rendering images Ra, Rb, Rc, and Rd as shown in FIG. 11 in time series based on the jaw movement-related data, and causes the generated simulation video to be displayed on the display 40, or transmits the data of the simulation video from the communication device 18 to an external device. Further, the data generation device 10 can display on the display 40 the result of dynamically simulating the jaw movement of the subject using only three-dimensional data in accordance with each frame image included in the video data acquired by the camera 20 based on the jaw movement-related data.

[0071] As described above, the data generation device 10 according to Embodiment 1 can generate jaw movement-related data of a subject by using video data obtained by photographing the subject while the subject is performing jaw movement with a part of the upper and lower dental arches exposed, and three-dimensional data of the upper and lower dental arches of the subject. In the simulation video based on the generated jaw movement-related data, the orientations of the upper jaw and the lower jaw and the positional relationship between the upper jaw and the lower jaw match accurately with the orientations of the upper jaw and the lower jaw and the positional relationship between the upper jaw and the lower jaw in the jaw movement of the subject shown by the frame images A to D constituting the video data. As a result, since the user does not need to have the patient wear an occlusal appliance or a jig, the preparation is not troublesome and no extra burden is imposed on the jaw movement of the subject. Therefore, the user can easily and accurately confirm the jaw movement of the subject.

[0072] The data generation device 10 first roughly searches for a rendering image including an upper and lower dental arch image that looks the same as the upper and lower dental arch images included in each frame image by pattern matching using a rendering image with the viewpoint tilted at a rough angle in the horizontal or vertical direction. Next, based on the rendering image generated (extracted) by the rough pattern matching, a rendering image with the viewpoint tilted at a fine angle in the horizontal or vertical direction is generated, and a rendering image including an upper and lower dental arch image that looks the same as the upper and lower dental arch images included in each frame image is searched in detail by pattern matching. For example, the data generation device 10 tries pattern matching between the upper and lower dental arch images of the frame image using rendering images with the viewpoint changed at rough angles (for example, at 3-degree intervals) in both the horizontal and vertical directions, and among the tried pattern matchings, based on a rendering image with a certain level of matching (equal to or higher than a first predetermined value), next, pattern matching is tried between the upper and lower dental arch images of the frame image using rendering images with the viewpoint changed at fine angles (for example, at 1-degree intervals) in both the horizontal and vertical directions. When a rendering image with a very high level of matching (equal to or higher than a second predetermined value, where the level of matching is higher than the first predetermined value) is obtained among the tried pattern matchings, it is considered that the pattern matching has been achieved. Thereby, the data generation device 10 can reduce the processing load imposed on the pattern matching and shorten the processing time of the pattern matching compared to performing detailed pattern matching from the beginning.

[0073] The video data acquired by the camera 20 may be input to the data generation device 10 in real time during the shooting of the subject. In this case, the data generation device 10 follows the input of the video data via the video data interface 13 during the shooting of the subject, and by executing the data generation process of FIG. 3, may generate jaw movement-related data in real time during the shooting. Thereby, since the jaw movement-related data is generated simultaneously with or immediately after the subject is photographed by the camera 20, the user can immediately show the simulation result to the subject, improving the convenience for the user.

[0074] <Embodiment 2> The data generation device 10 according to Embodiment 2 of the present disclosure will be described in detail with reference to FIG. 12. In the data generation device 10 according to Embodiment 2, only the parts different from the data generation device 10 according to Embodiment 1 will be described, and the same parts as the data generation device 10 according to Embodiment 1 will be denoted by the same reference numerals and their descriptions will not be repeated.

[0075] FIG. 12 is a flowchart for explaining an example of data generation processing executed by the data generation device 10 according to Embodiment 2. In the data generation processing according to Embodiment 2 shown in FIG. 12, first, jaw movement-related data corresponding to the first frame image included in the video data is extracted by pattern matching using only the first frame image, and then, an example of extracting jaw movement-related data corresponding to at least one other second frame image included in the video data using the jaw movement-related data corresponding to the first frame image will be described. In Embodiment 2, it is assumed that the position (viewpoint) of the camera 20 and the position of the object to be photographed are fixed while the subject is being photographed by the camera 20. That is, in each of the frame images A to D acquired by the camera 20, the position (viewpoint) of the camera 20 and the position of the upper and lower dental arches, which are the objects to be photographed, are the same as each other.

[0076] As shown in FIG. 12, the data generation device 10 extracts a plurality of frame images (for example, frame images A, B, C, D in FIG. 4) constituting the video data (S11), executes preprocessing (S12 to S14), and extracts upper and lower dental arch images corresponding to the upper and lower dental arches from each frame image (S15) in the same manner as the processes of S1 to S5 shown in FIG. 3.

[0077] Based on the three-dimensional data of the upper and lower dental arches acquired by the three-dimensional scanner 30, while generating a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions, the data generation device 10 roughly performs pattern matching between the upper and lower dental arch images included in the first frame image extracted in S15 and the upper and lower dental arch images included in each of the plurality of rendering images (S16). Furthermore, the data generation device 10 performs detailed pattern matching between the upper and lower dental arch images included in the first frame image extracted in S15 and the upper and lower dental arch images included in each of the plurality of rendering images (S17).

[0078] For example, in S16, the data generation device 10 roughly searches by pattern matching for a rendering image including an upper and lower dental arch image that looks the same as the upper and lower dental arch image included in the frame image A (the first frame image). Then, based on the rendering image generated (extracted) in S16, in S17, the data generation device 10 generates a rendering image with the viewpoint tilted at a fine angle in the horizontal or vertical direction, and searches in detail by pattern matching for a rendering image including an upper and lower dental arch image that looks the same as the upper and lower dental arch image included in the frame image A (the first frame image).

[0079] Based on the viewpoints of the camera 20 in the horizontal and vertical directions of the upper and lower dental arch images included in the first frame image extracted by the pattern matching in S16 and S17, the data generation device 10 generates a first rendering image corresponding to the upper and lower dental arch images included in the first frame image, and uses the generated first rendering image to generate a second rendering image corresponding to the upper and lower dental arch images included in another second frame image (S18).

[0080] For example, in S18, the data generation device 10 generates a rendering image Ra (first rendering image) corresponding to the upper and lower dental arch images included in the frame image A (first frame image), and uses the generated rendering image Ra (first rendering image) to generate rendering images Rb to Rd (second rendering images) corresponding to the upper and lower dental arch images included in each of the frame images B to D (second frame images).

[0081] Based on the generated rendering images Ra, Rb, Rc, and Rd, the data generation device 10 generates jaw movement-related data related to the jaw movement of the subject indicated by each of the frame images A, B, C, and D, and stores the jaw movement-related data in the storage device 12 in association with the data of each of the frame images A, B, C, and D (S20).

[0082] Based on the jaw movement-related data corresponding to the rendering images Ra, Rb, Rc, and Rd indicating the jaw movement of the subject stored in the storage device 12, the data generation device 10 generates a simulation result of the jaw movement and outputs the generated simulation result (S21).

[0083] As described above, the data generation device 10 according to the second embodiment first extracts the jaw movement-related data of the upper and lower dental arches corresponding to the first frame image by pattern matching using only the first frame image included in the video data, and then uses the jaw movement-related data of the upper and lower dental arches corresponding to the first frame image to extract the jaw movement-related data of the upper and lower dental arches corresponding to the second frame image included in the video data. Therefore, only one frame image is the target for executing pattern matching. As a result, the data generation device 10 can reduce the processing burden of pattern matching and shorten the processing time of pattern matching compared to performing pattern matching on each of the plurality of frame images constituting the video data.

[0084] <Embodiment 3> The data generation device 10 according to Embodiment 3 of the present disclosure will be described in detail with reference to FIGS. 13 to 16. In the data generation device 10 according to Embodiment 3, only the parts different from the data generation device 10 according to Embodiment 1 will be described, and the same parts as the data generation device 10 according to Embodiment 1 will be denoted by the same reference numerals and their description will not be repeated.

[0085] FIG. 13 is a flowchart for explaining an example of data generation processing executed by the data generation device 10 according to Embodiment 3. In the data generation processing according to Embodiment 3 shown in FIG. 13, an example will be described in which the regions (shapes) of the upper and lower dental arches in each frame image constituting the video data are estimated using AI (artificial intelligence), and further, each rendering image corresponding to the upper and lower dental arches included in each frame image is estimated using AI.

[0086] As shown in FIG. 13, the data generation device 10 extracts a plurality of frame images (for example, frame images A, B, C, D in FIG. 4) constituting the video data (S31) and executes preprocessing (S32 to S34) in the same manner as the processing of S1 to S4 shown in FIG. 3.

[0087] The data generation device 10 estimates the upper and lower dental arches in each frame image constituting the video data using AI (S35). Specifically, the data generation device 10 uses an estimation model 130 for estimating the regions (shapes) of the upper and lower dental arches in each frame image based on the input data including each frame image, and estimates the regions (shapes) of the upper and lower dental arches in each frame image based on the input data.

[0088] The estimation model 130 includes a known neural network used in image recognition processing by deep learning, such as a convolutional neural network (CNN), a generative adversarial network (GAN), a recurrent neural network (RNN), or a long short-term memory network (LSTM network), and parameters related to the neural network.

[0089] Here, with reference to FIG. 14, the supervised learning for the estimation model 130 will be described. FIG. 14 is a diagram for explaining the outline of the supervised learning executed by the data generation device 10 according to the third embodiment. As shown in FIG. 14, in the learning phase, the data generation device 10 executes the learning program 110 to machine-learn the estimation model 130 based on the learning data 120 including the input 1 and the input 2 (correct answer). In the utilization phase, the data generation device 10 uses the learned estimation model 130 to estimate the output based on the input 1.

[0090] FIG. 15 is a diagram for explaining the input and output of supervised learning in the data generation device 10 according to Embodiment 3. As shown in FIG. 15, as input 1 which is machine learning problem data, a frame image including an upper and lower dental arch image is used. As input 2 which is machine learning correct answer data, data for specifying the regions of the upper and lower dental arches in the frame image of input 1 is used. For example, as data for specifying the regions of the upper and lower dental arches in the frame image, data for distinguishing between the regions of the upper and lower dental arches and the regions of parts other than the upper and lower dental arches in the frame image can be cited. For example, a frame image with data of "1" assigned to the regions of the upper and lower dental arches and "0" assigned to the regions of parts other than the upper and lower dental arches is used as input 2. Further, data for specifying regions may be distinguished between the upper dental arch and the lower dental arch. For example, a frame image with data of "1" assigned to the region of the upper dental arch, "2" assigned to the region of the lower dental arch, and "0" assigned to the regions of parts other than the upper and lower dental arches may be used as input 2. As the output estimated by the data generation device 10 based on input 1, the estimation result of data for specifying the regions of the upper and lower dental arches in the frame image of input 1 is used.

[0091] That is, in the learning phase, the data generation device 10 estimates data (output) for specifying the regions of the upper and lower dental arches in the frame image based on a frame image (input 1) including an upper and lower dental arch image, and compares the estimated data (output) for specifying the regions of the upper and lower dental arches with the data (input 2) for specifying the regions of the upper and lower dental arches in the frame image which is the correct answer. If the two match, the parameters included in the estimation model 130 are not updated. On the other hand, if the two do not match, the parameters included in the estimation model 130 are updated so that the two match, thereby optimizing the estimation model 130. The learned estimation model 130 machine-learned in this way is stored in the storage device 12 as the data generation program 100. In the utilization phase, the data generation device 10 uses the learned estimation model 130 to estimate the upper and lower dental arch images (output) included in the frame image, that is, the shapes of the upper and lower dental arches indicated by the frame image, based on a frame image (input 1) including an upper and lower dental arch image.

[0092] Returning to FIG. 13, the data generation device 10 estimates, using AI, each rendering image (for example, the rendering images Ra, Rb, Rc, Rd in FIG. 11) corresponding to each upper and lower dental arch image (for example, the upper and lower dental arch images a, b, c, d in FIG. 5) estimated in S35 based on each frame image A, B, C, D constituting the video data (S36). Specifically, the data generation device 10 estimates, based on input data including the upper and lower dental arch images included in each frame image estimated in S35 and a plurality of rendering images including the upper and lower dental arch images, a rendering image corresponding to the upper and lower dental arch images included in each frame image using an estimation model 130 for estimating a rendering image corresponding to the upper and lower dental arch images based on the input data.

[0093] FIG. 16 is a diagram for explaining the input and output of supervised learning in the data generation device 10 according to Embodiment 3. As shown in FIG. 16, as input 1 which is machine learning problem data, an upper and lower dental arch image extracted based on each frame image and three-dimensional data of the upper and lower dental arches are used. As input 2 which is machine learning correct answer data, a rendering image corresponding to the upper and lower dental arch images included in each frame image of input 1 is used. As the output estimated by the data generation device 10 based on input 1, an estimation result of a rendering image corresponding to the upper and lower dental arch images included in each frame image of input 1 is used.

[0094] That is, in the learning phase, the data generation device 10 estimates a rendering image (output) corresponding to the upper and lower dental arch images included in each frame image based on the upper and lower dental arch images and the three-dimensional data of the upper and lower dental arches (input 1) extracted based on each frame image, compares the estimated rendering images (output) with the rendering images (input 2) corresponding to the upper and lower dental arch images included in each frame image that is the correct answer, and if they match, does not update the parameters included in the estimation model 130, while if they do not match, updates the parameters included in the estimation model 130 so that they match, thereby optimizing the estimation model 130. The learned estimation model 130 machine-learned in this way is stored in the storage device 12 as the data generation program 100. In the utilization phase, the data generation device 10 estimates a rendering image (output) corresponding to the upper and lower dental arch images included in each frame image based on the upper and lower dental arch images and the three-dimensional data of the upper and lower dental arches (input 1) extracted based on each frame image by using the learned estimation model 130.

[0095] Returning to FIG. 13, the data generation device 10 generates jaw movement-related data related to the jaw movement of the subject indicated by each frame image A, B, C, D based on the extracted rendering images Ra, Rb, Rc, Rd, and stores the jaw movement-related data in the storage device 12 in association with the data of each frame image A, B, C, D (S37).

[0096] The data generation device 10 generates a simulation result of the jaw movement based on the jaw movement-related data corresponding to the rendering images Ra, Rb, Rc, Rd indicating the jaw movement of the subject stored in the storage device 12, and outputs the generated simulation result (S38).

[0097] As described above, the data generation device 10 according to the third embodiment can extract upper and lower dental arch images from each frame image constituting the moving image data with higher accuracy in order to estimate the region (shape) where the upper and lower dental arches are located in each frame image constituting the moving image data using AI. Further, the data generation device 10 can extract each rendering image corresponding to the upper and lower dental arches included in each frame image with higher accuracy in order to estimate each rendering image corresponding to the upper and lower dental arches included in each frame image using AI. As a result, the data generation device 10 can generate jaw movement-related data that accurately matches the jaw movement of the subject.

[0098] <Embodiment 4> The data generation device 10 according to the fourth embodiment of the present disclosure will be described in detail with reference to FIGS. 17 to 19. In the data generation device 10 according to the fourth embodiment, only the parts different from the data generation device 10 according to the third embodiment will be described, and the same parts as those of the data generation device 10 according to the third embodiment will be denoted by the same reference numerals and their description will not be repeated.

[0099] In the data generation device 10 according to the fourth embodiment, the camera 20 is constituted by a three-dimensional camera capable of detecting the three-dimensional position coordinates of the imaging target. For example, the camera 20 is a ToF (Time of Flight) type depth camera capable of detecting the three-dimensional position coordinates of the imaging target by measuring the distance to the imaging target using the reflection of light (for example, infrared light), or a stereo camera capable of detecting the three-dimensional position coordinates of the imaging target by imaging the same space from different positions by a plurality of (for example, two) cameras. That is, the moving image data input to the data generation device 10 according to the fourth embodiment includes three-dimensional data that is the position information (coordinates of each axis in the vertical direction, horizontal direction, and height direction) of each point in a point cloud (a plurality of points) showing the surface of the upper and lower dental arches of the subject who is the imaging target.

[0100] FIG. 17 is a flowchart for explaining an example of data generation processing executed by the data generation device 10 according to Embodiment 4. As shown in FIG. 17, the data generation device 10 extracts a plurality of frame images (for example, frame images A, B, C, and D in FIG. 4) constituting the video data (S41) and executes preprocessing (S42 to S44) in the same manner as the processing of S1 to S5 shown in FIG. 3. However, in Embodiment 4, each frame image extracted in S41 is composed of video data including three-dimensional position coordinates.

[0101] The data generation device 10 estimates three-dimensional data of the upper and lower dental arches from each frame image constituting the video data using AI (S45). Specifically, the data generation device 10 uses an estimation model 130 for estimating three-dimensional data of the upper and lower dental arches included in each frame image based on input data including each frame image, and estimates the three-dimensional data of the upper and lower dental arches included in each frame image based on the input data.

[0102] FIG. 18 is a diagram for explaining the input and output of supervised learning in the data generation device 10 according to Embodiment 4. As shown in FIG. 18, as input 1 which is machine learning problem data, a frame image constituting three-dimensional video data including an upper and lower dental arch image is used. As input 2 which is machine learning correct answer data, three-dimensional data of the upper and lower dental arches included in the frame image of input 1 is used. As the output estimated by the data generation device 10 based on input 1, an estimation result of three-dimensional data of the upper and lower dental arches included in the frame image of input 1 is used.

[0103] That is, in the learning phase, the data generation device 10 estimates the three-dimensional data (output) of the upper and lower dental arches included in the frame image based on the frame image (input 1) constituting the three-dimensional moving image data including the upper and lower dental arch images, compares the estimated three-dimensional data (output) of the upper and lower dental arches with the three-dimensional data (input 2) of the upper and lower dental arches included in the frame image that is the correct answer, and if they match, does not update the parameters included in the estimation model 130, while if they do not match, updates the parameters included in the estimation model 130 so that they match, thereby optimizing the estimation model 130. The learned estimation model 130 machine-learned in this way is stored in the storage device 12 as the data generation program 100. In the utilization phase, the data generation device 10 estimates the three-dimensional data (output) of the upper and lower dental arches included in the frame image based on the frame image (input 1) constituting the three-dimensional moving image data including the upper and lower dental arch images, using the learned estimation model 130.

[0104] Returning to FIG. 17, the data generation device 10 estimates the three-dimensional data corresponding to the upper and lower dental arches extracted in S45 based on each of the frame images A, B, C, D constituting the moving image data from the three-dimensional data acquired by the three-dimensional scanner 30, using AI (S46). Specifically, the data generation device 10 estimates the three-dimensional data corresponding to the upper and lower dental arches included in each frame image based on the input data including the three-dimensional data of the upper and lower dental arches included in each frame image estimated in S35 and the three-dimensional data of the upper and lower dental arches acquired by the three-dimensional scanner 30, using the estimation model 130 for estimating the three-dimensional data corresponding to the upper and lower dental arches based on the input data.

[0105] FIG. 19 is a diagram for explaining the input and output of supervised learning in the data generation device 10 according to Embodiment 4. As shown in FIG. 19, as input 1 which is machine learning problem data, three-dimensional data of the upper and lower dental arches extracted based on each frame image and three-dimensional data of the upper and lower dental arches acquired by the three-dimensional scanner 30 are used. As input 2 which is the correct data of machine learning, three-dimensional data corresponding to the upper and lower dental arches included in each frame image of input 1 is used. As the output estimated by the data generation device 10 based on input 1, the estimation result of the three-dimensional data acquired by the three-dimensional scanner 30 corresponding to the upper and lower dental arches included in each frame image of input 1 is used.

[0106] That is, in the learning phase, based on the three-dimensional data of the upper and lower dental arches (input 1) extracted based on each frame image, the data generation device 10 estimates, from the three-dimensional data acquired by the three-dimensional scanner 30, the three-dimensional data corresponding to the upper and lower dental arches included in each frame image, and compares the estimated three-dimensional data (output) corresponding to the upper and lower dental arches with the three-dimensional data (input 2) of the upper and lower dental arches included in each frame image which is the correct answer. If the two match, the parameters included in the estimation model 130 are not updated. On the other hand, if the two do not match, the parameters included in the estimation model 130 are updated so that the two match, thereby optimizing the estimation model 130. The learned estimation model 130 machine-learned in this way is stored in the storage device 12 as the data generation program 100. In the utilization phase, the data generation device 10 uses the learned estimation model 130 to estimate, from the three-dimensional data of the upper and lower dental arches (input 1) extracted based on each frame image, the three-dimensional data (output) corresponding to the upper and lower dental arches included in each frame image from the three-dimensional data acquired by the three-dimensional scanner 30.

[0107] Returning to FIG. 17, the data generation device 10 generates jaw movement-related data related to the jaw movement of the subject indicated by each of the extracted frame images A, B, C, D, and stores the jaw movement-related data in the storage device 12 in association with the data of each of the frame images A, B, C, D (S47).

[0108] Based on the jaw movement-related data of the subject stored in the storage device 12, the data generation device 10 generates a simulation result of the jaw movement and outputs the generated simulation result (S48).

[0109] As described above, the data generation device 10 according to the fourth embodiment can extract the upper and lower dental arches from each frame image constituting the video data with higher accuracy because it estimates the three-dimensional data of the upper and lower dental arches from each frame image constituting the video data using AI. Further, the data generation device 10 can extract the three-dimensional data corresponding to the upper and lower dental arches included in each frame image with higher accuracy because it estimates the three-dimensional data corresponding to the upper and lower dental arches included in each frame image from the three-dimensional data acquired by the three-dimensional scanner 30 using AI. Moreover, since the video data acquired by the camera 20 is three-dimensional data including the position information of each point group indicating the surface of the upper and lower dental arches of the subject, the data generation device 10 can perform matching between the three-dimensional data of the upper and lower dental arches acquired by the camera 20 and the three-dimensional data of the upper and lower dental arches acquired by the three-dimensional scanner 30. Thereby, the data generation device 10 can generate jaw movement-related data that accurately matches the jaw movement of the subject and can shorten the processing time of the matching.

[0110] <Embodiment 5> The data generation device 10 according to the fifth embodiment of the present disclosure will be described in detail with reference to FIG. 20. In the data generation device 10 according to the fifth embodiment, only the parts different from the data generation device 10 according to the first embodiment will be described, and the same parts as the data generation device 10 according to the first embodiment will be denoted by the same reference numerals and the description thereof will not be repeated.

[0111] FIG. 20 is a flowchart for explaining an example of data generation processing executed by the data generation device 10 according to Embodiment 5. In Embodiment 5 shown in FIG. 20, the data generation system 1 includes a CT (Computed Tomography) imaging device (not shown). The CT imaging device is an X-ray imaging device that performs computed tomography of the upper and lower jaws of a subject by rotating an X-ray transmitter and receiver, which are a type of radiation, around the subject's face. The user can obtain three-dimensional volume (voxel) data of hard tissue parts (bones, teeth, etc.) other than soft tissue parts (skin, gums, etc.) around the upper and lower jaws of the subject by photographing the upper and lower jaws of the subject using the CT imaging device. The user can generate a tomographic image or an appearance image of the imaging target by tomographically processing the volume data of the imaging target obtained by the CT imaging device. Hereinafter, the three-dimensional surface coordinate data acquired by the three-dimensional scanner 30 is also referred to as "IOS data", and the three-dimensional volume data acquired by the CT imaging device is also referred to as "CT data". Since both IOS data and CT data correspond to three-dimensional data of the upper and lower dental arches, IOS data and CT data can also be collectively referred to as "three-dimensional data".

[0112] In the data generation process according to Embodiment 5, in addition to the IOS data of the upper and lower dental arches of the subject acquired by the three-dimensional scanner 30, CT data of the upper and lower jaws including the upper and lower dental arches of the subject acquired by the CT imaging device is input.

[0113] As shown in FIG. 20, the data generation device 10 generates surface shape data indicating the surface shape of the hard tissue parts (bones, teeth, etc.) in the upper and lower jaws of the subject based on the CT data input from the CT imaging device (S51). The data generation device 10 separates the surface shape data generated in S51 between the upper jaw and the lower jaw (S52).

[0114] The data generation device 10 generates a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions based on the IOS data of the upper and lower dental arches of the subject acquired by the three-dimensional scanner 30 and the surface shape data of the upper and lower jaws generated in S52. Since the IOS data includes the position information of each point in the point cloud indicating the surface of the upper and lower dental arches, it can accurately represent the position of each tooth and the occlusion between the upper dental arch and the lower dental arch. On the other hand, the surface shape data of the upper and lower jaws generated based on the CT data can accurately represent the shape of the surface of the roots of the upper and lower dental arches and each tooth. Therefore, the data generation device 10 can generate a rendering image that more accurately represents the shape and position of the two-dimensional upper and lower dental arches by combining the IOS data of the upper and lower dental arches and the surface shape data of the upper and lower jaws.

[0115] While generating a plurality of rendering images including two-dimensional upper and lower dental arches viewed from multiple directions based on the rendering images using the IOS data and the CT data as described above, the data generation device 10 roughly performs pattern matching between the upper and lower dental arch images included in each frame image extracted in S5 and the upper and lower dental arch images included in each of the plurality of rendering images (S53), and further performs detailed pattern matching between the upper and lower dental arch images included in each frame image extracted in S5 and the upper and lower dental arch images included in each of the plurality of rendering images (S54).

[0116] Thereafter, the data generation device 10 can generate subject jaw movement-related data by executing the processes of S8 to S10 in the same manner as the data generation process according to Embodiment 1 shown in FIG. 3.

[0117] As described above, the data generation device 10 according to Embodiment 5 can generate a rendering image that more accurately represents the shape and position of the upper and lower dental arches in two dimensions, using not only the IOS data acquired by the three-dimensional scanner 30 but also the CT data acquired by the CT imaging device, and can generate subject jaw movement-related data based on the video data of the subject acquired by the camera 20 and the rendering image. Thereby, the data generation device 10 can generate jaw movement-related data that accurately matches the jaw movement of the subject.

[0118] Note that in the data generation device 10 according to either Embodiment 2 or Embodiment 3, as the three-dimensional data input to the data generation device 10, the IOS data acquired by the three-dimensional scanner 30 and the CT data acquired by the CT imaging device can be adopted. In this case, in the data generation device 10 according to either Embodiment 2 or Embodiment 3, the processes S51 to S54 of the data generation process according to Embodiment 5 shown in FIG. 20 may be executed to perform pattern matching. Further, the three-dimensional data input to the data generation device 10 is not limited to the IOS data obtained by directly scanning the upper and lower dental arches of the subject using the optical method of the three-dimensional scanner 30 as described above, and the CT data obtained by performing computed tomography of the upper and lower jaws of the subject using the CT imaging device, but may also include surface shape data obtained by scanning a dental and maxillofacial model created from an impression of the upper and lower dental arches of the subject using a desktop scanner.

[0119] Incidentally, the data generation device 10 was configured to separate the surface shape data between the upper jaw and the lower jaw, and generate a rendering image corresponding to the upper jaw and a rendering image corresponding to the lower jaw based on the separated surface shape data of the upper jaw and the lower jaw, respectively. However, a rendering image of the upper and lower jaws may be generated based on the surface shape data of the upper and lower jaws combined. Then, the data generation device 10 may perform pattern matching with the upper and lower dental arch images using the generated rendering image of the upper and lower jaws. Further, in the video data of the biting motion, the occluded state is also captured. Therefore, when generating the rendering image of the upper and lower jaws, the data generation device 10 may additionally use the IOS data in the occluded state.

[0120] <Embodiment 6> The data generation device 10 according to Embodiment 6 of the present disclosure will be described in detail with reference to FIG. 21. In the data generation device 10 according to Embodiment 6, only the parts different from the data generation device 10 according to Embodiment 1 will be described, and the same parts as the data generation device 10 according to Embodiment 1 will be denoted by the same reference numerals and the description thereof will not be repeated.

[0121] In the data generation device 10 according to Embodiment 6, the camera 20 is configured to capture a moving image of the imaging target (for example, the face of the subject) from a plurality of viewpoints. For example, FIG. 21 is a diagram showing an application example of the data generation system 1 and the data generation device 10 according to Embodiment 6. As shown in FIG. 21, the data generation system 1 according to Embodiment 6 includes a camera 20A that captures the face of the subject from substantially in front of the subject (for example, the front of the face of the subject), and a camera 20B that captures the face of the subject from substantially the side of the subject (for example, the side of the face of the subject).

[0122] The data generation device 10 may perform pattern matching (for example, the processes of S6 and S7 in FIG. 3) using the upper and lower dental arch images extracted from each frame image constituting the video data acquired by the camera 20A and the upper and lower dental arch images extracted from each frame image constituting the video data acquired by the camera 20B. At this time, the data generation device 10 may preferentially use, for pattern matching, the video data with higher pattern matching accuracy among the video data acquired by the camera 20A and the video data image acquired by the camera 20B.

[0123] For example, the camera 20A photographs the face of the subject from the front of the face of the subject, and the shape of the upper and lower dental arches in the horizontal direction extracted from each frame image is likely to be accurately represented. Therefore, the data generation device 10 may preferentially use the upper and lower dental arch images extracted from each frame image constituting the video data acquired by the camera 20A to perform pattern matching in the horizontal direction.

[0124] Also, the camera 20B photographs the face of the subject from the side of the face of the subject, and the shape of the upper and lower dental arches in the vertical direction extracted from each frame image is likely to be accurately represented. Therefore, the data generation device 10 may preferentially use the upper and lower dental arch images extracted from each frame image constituting the video data acquired by the camera 20B to perform pattern matching in the vertical direction.

[0125] As described above, since the data generation device 10 according to the sixth embodiment can generate jaw movement-related data of the subject using the video data obtained by photographing the subject from a plurality of viewpoints, it can generate jaw movement-related data that accurately matches the jaw movement of the subject.

[0126] Note that the camera is not limited to being disposed substantially in front of or substantially to the side of the subject, and may be disposed substantially diagonally in front of the subject. That is, the data generation system 1 may include one camera disposed at any one of a position substantially in front of the subject, a position substantially diagonally in front of the subject, and a position substantially to the side of the subject, or may include a plurality of cameras disposed at any two or more of the positions substantially in front of the subject, a position substantially diagonally in front of the subject, and a position substantially to the side of the subject. The data generation device 10 may acquire moving image data from a plurality of cameras that simultaneously photograph the subject from a plurality of viewpoints, and execute pattern matching (for example, the processes of S6 and S7 in FIG. 3) using the moving image data acquired from the plurality of cameras.

[0127] Note that the data generation device 10 according to each of the above-described Embodiments 1 to 6 may be a cloud-type server device. For example, a user or the subject himself / herself may upload and register moving image data obtained by photographing the subject and three-dimensional data of the upper and lower dental arches of the subject to the cloud-type data generation device 10. Then, the data generation device 10 may generate and distribute jaw movement-related data based on the moving image data and the three-dimensional data, so that the user or the subject can view or save a simulation video of the jaw movement that can be generated based on the jaw movement-related data.

[0128] Note that the data generation system 1 and the data generation device 10 according to each of the above-described Embodiments 1 to 6 may include the configurations and functions of each other alone or in combination.

[0129] The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is shown by the claims rather than the above description, and it is intended that all modifications within the meaning and scope equivalent to the claims are included. Note that the configurations exemplified in the present embodiment and the configurations exemplified in the modification examples can be combined as appropriate.

Description of Reference Numerals

[0130] 1 data generation system, 10 data generation devices, 11 arithmetic units, 12 storage devices, 13 video data interfaces, 14 three-dimensional data interfaces, 15 display interfaces, 16 peripheral device interfaces, 17 media readers, 18 communication devices, 20, 20A, 20B cameras, 30 three-dimensional scanners, 40 displays, 51 keyboards, 52 mice, 60 removable disks, 100 data generation programs, 110 learning programs, 120 learning data, 130 estimation models.

Claims

1. A data generation device that generates jaw movement-related data related to the jaw movement of a subject, an input unit to which video data obtained by photographing the subject who is making a jaw movement with a part of the upper and lower dental arches exposed from the front of the subject or from a viewpoint inclined by a predetermined angle in the horizontal or vertical direction from the front using one camera, and three-dimensional data of the upper and lower dental arches of the subject are input; a calculation unit that generates the jaw movement-related data related to the jaw movement of the subject by performing pattern matching between the part of the upper and lower dental arches included in each of the plurality of frame images constituting the video data and the upper and lower dental arches included in a rendering image generated based on the three-dimensional data of the upper and lower dental arches, and searching for the three-dimensional position of the three-dimensional data of the upper and lower dental arches during the jaw movement; The calculation unit generates a plurality of rendering images at a viewpoint inclined by a predetermined angle in the horizontal or vertical direction from the front of the subject based on the three-dimensional data of the upper and lower dental arches. A data generation device.

2. The three-dimensional data input to the input unit includes at least one of three-dimensional data including position information of each point group indicating the surface of the upper and lower dental arches and three-dimensional data obtained by computer tomography of the upper and lower dental arches. The data generation device according to claim 1.

3. The calculation unit displays, on a display unit, a result of dynamically simulating the jaw movement of the subject using only the three-dimensional data in accordance with each of the plurality of frame images based on the jaw movement-related data. The data generation device according to claim 1 or claim 2.

4. The video data includes video data obtained by photographing using at least one of a monocular camera and a depth camera. The data generation device according to claim 1 or claim 2.

5. The data generation device according to claim 1 or claim 2, wherein the calculation unit generates the jaw movement-related data following the input of the video data to the input unit during photographing of the subject.

6. The data generation device according to claim 1, wherein the calculation unit estimates a rendering image corresponding to the partial upper and lower dental arches included in the frame image, using an estimation model for estimating a rendering image corresponding to the partial upper and lower dental arches included in the frame image, based on input data including the shape of the partial upper and lower dental arches included in the frame image constituting the video data and the three-dimensional data of the upper and lower dental arches.

7. The data generation device according to claim 1, wherein the calculation unit estimates a region of the partial upper and lower dental arches in the frame image, using an estimation model for estimating a region of the partial upper and lower dental arches in the frame image, based on input data including the frame image constituting the video data.

8. The data generation device according to claim 1, wherein the calculation unit estimates the three-dimensional data of the upper and lower dental arches, using an estimation model for estimating the three-dimensional data of the upper and lower dental arches corresponding to the partial upper and lower dental arches included in the frame image, based on input data including the shape of the partial upper and lower dental arches included in the frame image constituting the video data and the three-dimensional data of the upper and lower dental arches.

9. A data generation method for generating jaw movement-related data related to the jaw movement of a subject by a computer, comprising: a step of inputting video data obtained by photographing the subject who is making a jaw movement with a part of the upper and lower dental arches exposed, from the front of the subject or from a viewpoint tilted by a predetermined angle in a horizontal or vertical direction from the front, using one camera, and three-dimensional data of the upper and lower dental arches of the subject; A step of generating the jaw movement-related data related to the jaw movement of the subject by performing pattern matching between the partial upper and lower dental arches included in each of the plurality of frame images constituting the video data and the upper and lower dental arches included in a rendering image generated based on the three-dimensional data of the upper and lower dental arches, and searching for the three-dimensional position of the three-dimensional data of the upper and lower dental arches during the jaw movement. The generating step includes a step of generating a plurality of rendering images at a viewpoint inclined by a predetermined angle in a horizontal direction or a vertical direction from the front of the subject based on the three-dimensional data of the upper and lower dental arches. A data generation method.

10. A data generation program for generating jaw movement-related data related to the jaw movement of a subject, Causing a computer to Video data obtained by photographing the subject who is making a jaw movement with a part of the upper and lower dental arches exposed using one camera at the front of the subject or at a viewpoint inclined by a predetermined angle in a horizontal direction or a vertical direction from the front, and the three-dimensional data of the upper and lower dental arches of the subject are input steps, Causing the computer to perform pattern matching between the partial upper and lower dental arches included in each of the plurality of frame images constituting the video data and the upper and lower dental arches included in a rendering image generated based on the three-dimensional data of the upper and lower dental arches, and searching for the three-dimensional position of the three-dimensional data of the upper and lower dental arches during the jaw movement, thereby generating the jaw movement-related data related to the jaw movement of the subject. The generating step includes a step of generating a plurality of rendering images at a viewpoint inclined by a predetermined angle in a horizontal direction or a vertical direction from the front of the subject based on the three-dimensional data of the upper and lower dental arches. A data generation program.

Images