Image processing device, imaging device, imaging system, and method

Abstract

To make a plurality of captured intraoral images look the same as an appearance when a patient is observed.SOLUTION: In an imaging device, intraoral images are captured, and portion information indicating a portion in the oral cavity and mirror use information indicating presence or absence of use of a mirror in capturing the images are set. An image file is generated by adding the portion information and the mirror use information set in capturing the images to image data on the respective captured intraoral images. An image processing device determines whether or not it is necessary to execute at least one of right-left inversion processing and vertical rotation processing on the basis of the portion information and the mirror use information for the image data on the image file generated by the imaging device, and executes the processing that it determines as necessary.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, an imaging system, and a method, and particularly relates to a technique for processing an intraoral image captured.

Background Art

[0002] In oral surgery and dentistry, medical staff take intraoral photographs for the purpose of diagnosing and recording the inside of the mouth. By using the captured intraoral photographs, it becomes possible to establish treatment policies and plans, and to evaluate after treatment procedures. At that time, there are methods such as the five-shot method, nine-shot method, and twelve-shot method for taking pictures of the patient's oral cavity from multiple directions. Among them, the five-shot method is widely used as a method for recording the state of the oral cavity with a small number of shots. Visible light intraoral images have advantages such as being able to diagnose the state of the gums and the color of the teeth, and being used for recording.

[0003] On the other hand, as another method for diagnosing and recording the inside of the mouth, X-ray photographs may also be used. Patent Document 1 discloses an image processing apparatus having an image size adjustment unit and an image brightness adjustment unit for adjusting the size and brightness when combining and synthesizing a plurality of X-ray photographs taken from multiple directions, and adjusting the brightness along the average of the histogram.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique disclosed in Patent Document 1, although the brightness and size can be made uniform for X-ray photographs, it is not possible to make the intraoral photographs taken in a different shooting order or taken with a mirror the same as the way it looks when observing the patient.

[0006] This invention was made in view of the above-mentioned problems, and aims to make multiple intraoral images captured by the patient the same as what is seen when the patient is observed. [Means for solving the problem]

[0007] To achieve the above objective, the image processing apparatus of the present invention includes an acquisition means for acquiring an image file that includes a first image data of an intraoral image, location information indicating a location within the oral cavity, and mirror usage information indicating whether or not a mirror was used during shooting; and a processing means for determining whether or not at least one of left-right inversion processing and up-down rotation processing is necessary for the first image data of the acquired image file based on the location information and the mirror usage information, and performing the processing that is determined to be necessary. [Effects of the Invention]

[0008] According to the present invention, it is possible to make multiple intraoral images captured and the appearance when observing the patient the same. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the device configuration of a dental system in an embodiment of the present invention. [Figure 2] A block diagram showing the hardware configuration of the image processing device in the embodiment. [Figure 3] A block diagram showing the schematic functional configuration of a digital still camera in an embodiment. [Figure 4] A schematic rear view of a digital still camera in an embodiment. [Figure 5] A flowchart illustrating the imaging control process in the first embodiment. [Figure 6] A flowchart illustrating image processing based on supplementary information in the first embodiment. [Figure 7] A diagram showing an example of the user interface of the image processing device in the first embodiment. [Figure 8] A flowchart illustrating the image region estimation process and image processing in the second embodiment. [Figure 9] A conceptual diagram illustrating the possible arrangement of captured images in the second embodiment. [Figure 10] A diagram showing a table of inference results for the state of each tooth based on images of the occlusal view in the second embodiment. [Figure 11] This figure shows a table of inference results regarding the condition of each tooth based on a lateral view image assuming no mirror is used in the second embodiment. [Figure 12] This figure shows a table of inference results regarding the condition of each tooth based on a lateral view image assuming the use of a mirror in the second embodiment. [Figure 13] A diagram showing a table of inference patterns for image placement based on the state of each tooth in the second embodiment. [Figure 14] A diagram showing an example of an image taken in a modified case. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] <First Embodiment> [Definition of terms] In this embodiment, the intraoral photographs are visible light image data acquired using the five-image method in dental diagnosis. The five images in the five-image method consist of a frontal view, an occlusal view of the maxilla, an occlusal view of the mandible, a left lateral view, and a right lateral view. In this embodiment, the occlusal views of the maxilla and mandible together are referred to as the occlusal view. The left and right lateral views together are referred to as the lateral view. Furthermore, in this embodiment, the intraoral region refers to one of the following: the front, maxilla, mandible, left side, or right side.

[0012] The tooth formula number in this embodiment is a symbol for expressing the position of teeth in the tooth formula, and is assigned in ascending order from the front to the back for each of the upper, lower, left, and right sides. In the case of permanent teeth, the central incisor is 1 and the third molar is 8. In the case of deciduous teeth, the primary central incisor is A and the second deciduous molar is E. In the present invention, the tooth formula number refers to a name such as "the 6th tooth on the upper right". On the other hand, the tooth type is a name based on the shape of the tooth. For each of the upper, lower, left, and right sides, in order from the frontmost, they are the central incisor, lateral incisor, canine tooth, (first, second) premolar, (first, second, third) molar. In the case of deciduous teeth, they are the primary central incisor, primary lateral incisor, primary canine tooth, (first, second) deciduous molar. The tooth formula number specifies a single tooth and is described separately from the tooth type.

[0013] Also, in this embodiment, the teeth from the 1st to the 8th located on the right side of the upper jaw are represented as uR1 to uR8, and the teeth from the 1st to the 8th located on the left side of the upper jaw are represented as uL1 to uL8. Also, the teeth from the 1st to the 8th located on the right side of the lower jaw are represented as bR1 to bR8, and the teeth from the 1st to the 8th located on the left side of the lower jaw are represented as bL1 to bL8.

[0014] The tooth condition in this embodiment indicates the health condition of the teeth, the presence or absence of dental treatment, etc. For each tooth, at least one is selected from sound teeth, caries, filling, crown, implant, etc. The dental information in this embodiment is information that associates the tooth formula number, the condition of each tooth, and the image data for each patient.

[0015] [Prerequisite Conditions for Intraoral Images] For intraoral images taken using the five-image method, the images should match the view observed when observing the patient. For the maxillary occlusal view, the dental arch should be convex. For the mandibular occlusal view, the dental arch should be concave. For the frontal view, the patient's nose should be upward and the patient's jaw downward. For the left lateral view, the patient's nose should be upward and the patient's jaw downward, as in the frontal view, plus the molars should be on the right and the anterior teeth on the left. For the right lateral view, the patient's nose should be upward and the patient's jaw downward, as in the frontal view, plus the molars should be on the left and the anterior teeth on the right.

[0016] In the five-image method, the general procedure for taking intraoral images is as follows: For frontal images, the photographer faces the patient directly without using a mirror. For occlusal images, a mirror is usually used because it is impossible to photograph the molars without one. When the occlusal surface is photographed with the patient lying down and a mirror is used, the dental arches of both the upper and lower jaws generally appear convex. For lateral images, a mirror may be used, or the mouth may be widened using a corner retractor without a mirror. When a mirror is used, the image will be reversed left to right compared to how the patient's oral cavity appears when observed without a mirror.

[0017] Furthermore, for the same patient, the use of a mirror should not differ between the left and right sides during a series of image acquisitions; for example, a mirror should not be used only for the right lateral view while one is not used for the left lateral view.

[0018] [System Configuration] Figure 1 is a block diagram showing the device configuration of the dental system as the imaging system of this embodiment.

[0019] The imaging device 108 captures images of the oral cavity 109 of the patient being examined and generates visible light image data. In the following description, the images captured by the imaging device 108 will be described as visible light image data. Patient information such as the patient ID is added to the captured image data and transmitted to the image processing device 101, and captured images are deleted at the request of the image processing device 101. The imaging device 108 is also assumed to be equipped with input / output devices (not shown) for confirming captured images and selecting patient information, similar to commercially available digital cameras.

[0020] The image processing device 101 determines the dental formula number and the condition of the teeth based on information including image data transmitted from the imaging device 108. Alternatively, the process of adding information about the oral cavity region (front view, maxilla, mandible, left view, right view) of the oral image, which is performed by the dental electronic medical record display terminal 103 described later, may be performed in the image processing device 101 using the method for determining the dental formula number and the condition of the teeth.

[0021] Furthermore, the image processing device 101 communicates with the imaging device 108 and the dental electronic medical record system 104. When the imaging device 108 is powered on or the patient information is reconfirmed, the image processing device 101 obtains the patient information of the patient being examined from the dental electronic medical record system 104 and transmits it to the imaging device 108.

[0022] The dental electronic medical record display terminal 103 communicates with the dental electronic medical record system 104 to display and accept input during the creation of dental electronic medical records, and to perform processing to add information about the oral cavity area (front view, upper jaw, lower jaw, left view, right view) of oral images.

[0023] The dental electronic medical record system 104 receives a request from the dental electronic medical record display terminal 103, communicates with the image processing device 101, the in-hospital system 106, and the dental information DB 105, and sends and receives the necessary data associated with the dental electronic medical record creation process. The dental information database 105 communicates with the dental electronic medical record system 104 and stores, transmits, and receives patient dental information in association with patient information such as patient ID. The in-hospital system 106 communicates with the patient information database 107 to register and retrieve patient information, and also communicates with the dental electronic medical record system 104 to transmit patient information. The patient information database 107 communicates with the in-hospital system 106 to store and transmit / receive patient information. As a result, the dental electronic medical record system 104 obtains information on patients to be examined from the patient information database 107 via the in-hospital system 106.

[0024] The image processing device 101, imaging device 108, dental electronic medical record system 104, and in-hospital system 106 communicate via network 102. Communication between the dental electronic medical record display terminal 103 and the dental electronic medical record system 104 is performed via an HDMI® cable or the like. Communication between the dental information database 105 and the dental electronic medical record system 104, and between the in-hospital system 106 and the patient information database 107, is performed via a USB cable or the like. While the above example uses a network, HDMI, and USB as communication means, the present invention is not limited to these, and communication may, for example, be performed via wireless communication.

[0025] [A computer that operates as an image processing device] Figure 2 is a block diagram showing the hardware configuration of the image processing device 101 in this embodiment. The image processing device 101 has a processor (CPU) 201 that executes programs, and a ROM (Read Only Memory) 202 that stores programs. Furthermore, it has a RAM (Random Access Memory) 203 that loads data necessary when executing programs. It also has an HDD (Hard Disk Drive) 205 that stores inference data, inference results, data for generating inference data (e.g., training data), etc. Furthermore, it includes an input device 206 used when registering setting information for programs, a display 204 for display confirmation, an I / F (Interface) 207 used for communication with external systems, and a bus 208.

[0026] Each function in the image processing device 101 is realized by loading a predetermined program onto hardware such as the CPU 201 and ROM 202, and having the CPU 201 perform calculations. It is also realized through communication with an external system via the I / F 207, and by reading and writing data to the RAM 203 and HDD 205, with the CPU 201 performing calculations.

[0027] [A digital still camera that functions as an imaging device] Figure 3 is a block diagram showing the functional configuration of a digital still camera as an example of the imaging device 108 in this embodiment. By executing a predetermined control program in the digital still camera, the imaging process described below is realized, and the imaging device 108 functions.

[0028] The imaging unit 300 converts the incident optical image into an electrical signal using a solid-state image sensor, and generates image data by converting the obtained electrical signal from analog to digital.

[0029] The CPU 301 controls the entire digital still camera. The ROM 302 stores the operating procedures of the CPU 301 (for example, programs for processing when the digital still camera is powered on and basic input / output processing). The RAM 303 functions as the main memory of the CPU 301, and various programs, including control programs for implementing the processes described later, are loaded from the ROM 302 and executed by the CPU 301. The RAM 303 also provides a work area for the CPU 301 when it executes various processes.

[0030] The display device 304 performs various displays under the control of the CPU 301. For example, it displays data stored on a storage medium (not shown). The input device 305 consists of one or more combinations of switches, dials, touch panels, eye-tracking pointing devices, voice recognition devices, etc., for performing various operations, and includes, for example, a shutter release button located on the top of a digital still camera.

[0031] The media drive 306 allows for the insertion of removable storage media, enabling the storage of data on the storage media and the reading of data stored on the storage media. The network interface 307 is connected to the network 102 via a communication line 309. Data is sent and received through this network interface 307 to the image processing device 101 and to server computers and personal computers (not shown). The system bus 308 consists of an address bus, a data bus, and a control bus, and connects the aforementioned units.

[0032] The image processing unit 311 performs image processing on the image data output from the imaging unit 300. The CPU 301 temporarily stores the image data and attribute information generated by the imaging unit 300 in the RAM 303. Then, as needed, the image processing unit 311 performs a series of image processing operations to create image data that matches the characteristics of human vision.

[0033] In the file generation unit 312, based on the control of the CPU 301, the image data processed by the image processing unit 311 and stored in the RAM 303 is converted into image data in a general-purpose still image format, such as a JPEG image.

[0034] [User interface for imaging device] Figure 4 is a schematic rear view of a digital still camera operating as the imaging device 108 in this embodiment.

[0035] In Figure 4, the power button 401 switches the power ON / OFF. When the photographer presses the button while the digital still camera is not powered on, the CPU 301 determines that the photographer has instructed the camera to power on and turns the power on. When the photographer presses the button while the power is on, the CPU 301 determines that the photographer has instructed the camera to turn the power off and turns the power off.

[0036] When the power is turned on, display on the display 408 begins. Figure 4 shows an example of the screen for selecting the patient's body part to be the subject in this embodiment (hereinafter referred to as the "body part selection screen").

[0037] When the photographer presses the shutter release button 402, the CPU 301 determines that a still image capture command has been issued.

[0038] 403 is the up button, 404 is the right button, 405 is the down button, 406 is the left button, and 407 is the select button, forming the input device 305. When the photographer presses any of the direction buttons 403 to 406, the CPU 301 determines that the photographer has performed an operation to switch the selected object, and switches the selected object on the display 408 according to the direction of the pressed direction button 403 to 406. When the photographer presses the select button 407, the CPU 301 determines that the photographer has performed a confirmation operation, and stores the selected information in the RAM 303 and switches the state of the digital still camera.

[0039] The display 408 constitutes the display device 304 and, together with the touch panel laid on its surface, constitutes the input device 305. When the photographer touches any point on the screen with their finger, the CPU 301 determines that there has been an input instruction from the photographer, determines the content of the operation from the content displayed at the touched position, and performs various processing according to the content of the operation.

[0040] 409-422 refer to the content displayed on display 408, and here they show the content when the area selection screen is displayed as described above. Area 409 displays text prompting the photographer to select one of the options. Focus frame 410 is a frame that informs the photographer of the selected item. Selection item display area 411 is an area that lists the candidate options, and here it displays the front view 413, maxilla 414, mandible 415, left view 416, and right view 417, which are candidate areas when taking intraoral photographs. Scroll bar 412 switches the display area when the selection item display area 411 cannot display all the candidate options.

[0041] When the photographer presses a direction button 403 to 406, the CPU 301 determines that the photographer has made a selection change operation and moves the focus frame 410 within the selection item display area 411. Alternatively, the photographer can move the focus frame 410 by touching the area of ​​the desired selection item displayed on the display 408.

[0042] Menus 418, 419, 420, 421, and 422 are menus that associate whether or not to use a mirror when photographing each body part, and are hereinafter referred to as the "Mirror Usage Information Setting Menu". The initial value is unset, and when one of the Mirror Usage Information Setting Menus 418, 419, 420, 421, or 422 is selected using the focus frame 410, a pull-down menu will be displayed, for example, allowing the user to select whether or not to use a mirror. Alternatively, for example, when one of the Mirror Usage Information Setting Menus 418, 419, 420, 421, or 422 is selected using the focus frame 410, pressing the OK button 407 will cycle through the options, and the selection method is not restricted.

[0043] In the example shown in Figure 4, the mirror usage information setting menu 418 indicates that a mirror should not be used when photographing the front view. The mirror usage information setting menu 419 indicates that a mirror should be used when photographing the upper jaw, and the mirror usage information setting menu 420 indicates that a mirror should be used when photographing the lower jaw. The mirror usage information setting menus 421 and 422 indicate that the use of a mirror when photographing the left and right sides is not set.

[0044] When the focus frame 410 is in one of the mirror usage information setting menus 418 to 422, and a selection is confirmed by pressing the confirm button 407 or by moving to another mirror usage information setting menu 418 to 422, the CPU 301 determines the selected option.

[0045] Information regarding whether or not to use a mirror should be stored as mirror usage information, associated with information about each body part, on a non-volatile recording medium via RAM 303 and media drive 306. The purpose of recording mirror usage information on a non-volatile recording medium is that if the body parts that use mirrors are consistent across the medical institution, it eliminates the need to set the information each time the power is turned on or off. However, it is not always necessary to record mirror usage information; it can also be configured to be set each time an image is taken.

[0046] [Image control processing] Figure 5 is a flowchart showing the imaging control process in this embodiment. Here, we will describe a method for recording body part information and mirror usage information attached to image data in the imaging device 108.

[0047] In S501, the CPU 301 acquires patient information from the image processing device 101. In this embodiment, patient information refers to the patient's name, gender, and age at the time of the examination.

[0048] In S502, the CPU 301 displays the patient information received in S501 and prompts the user to confirm whether it matches the patient being examined. If it determines that the patient information does not match, the system returns to S501 and retrieves the patient information again from the image processing device 101. If it determines that the patient information matches, the system proceeds to S503. The patient information that is determined to match is stored in the RAM 303.

[0049] In S503, the CPU 301 displays the body part selection screen shown in Figure 4. In S504, the CPU 301 determines whether a body part has been selected. Whether a body part has been selected is determined by whether the confirm button 407 is pressed while the focus frame 410 is in one of the body parts 413 to 417. If it is determined that a body part has been selected, the CPU 301 stores the selected body part information as selected body part information in the RAM 303 and proceeds to S505. At this time, the mirror usage information associated with the selected body part information is also stored in the RAM 303. If it is not determined that a body part has been selected, the screen in S503 waits for input from the photographer.

[0050] In S505, the CPU 301 determines whether the mirror usage information stored in RAM 303 in S504 has been set. If either "mirror used" or "mirror not used" is set in the mirror usage information setting menu, it determines that the mirror usage information has been set and proceeds to S507. If it is not set in the mirror usage information setting menu, it determines that it has not been set and proceeds to S506. In S506, the CPU 301 displays a warning on display 408, for example, "Please set whether to use the mirror or not," and returns to S503.

[0051] In S507, the CPU 301 transitions to a state where imaging can be performed by the imaging unit 300 (imaging mode). In S508, the CPU 301 determines whether or not the shutter release button 402 has been pressed. If the shutter release button 402 has been pressed, the CPU 301 determines that the photographer has given an instruction to take an image and proceeds to S509. If the shutter release button 402 has not been pressed, the CPU 301 determines that the photographer has not yet given an instruction to take an image and returns to S507.

[0052] In step S509, the CPU 301 controls the imaging unit 300 to perform imaging and acquire image data.

[0053] In step S510, the CPU 301 controls the image processing unit 311 and the file generation unit 312 to convert the image data obtained in step 509 into image data in a general-purpose file format. Then, patient information, shooting date and time information, selected area information, and mirror usage information stored in the RAM 303 are associated with the image data and recorded as supplementary information. In this embodiment, JPEG is assumed to be an example of a general-purpose file format, and the supplementary information is recorded as header information of the JPEG file.

[0054] In S511, the CPU 301 displays the image taken in S509 and the OK button on the display 408 to confirm with the photographer whether the desired photo was taken. In S512, the CPU 301 determines whether the OK button was pressed by the photographer. The OK button press is determined by touch operation on the display 408 or by pressing the OK button 407. If it is determined that the OK button was pressed, the process proceeds to S513. On the other hand, if it is determined that the OK button has not been pressed for a predetermined time, the process returns to S507 and prompts the photographer to retake the photo. Alternatively, an NG button may also be displayed on the display 408, and the process may return to S507 if the NG button is pressed.

[0055] In S513, the CPU 301 transmits the image file generated in S510 to the image processing device 101 via the network 102. It also stores the selected area information in the RAM 303 as already captured area information, associating it with the patient information.

[0056] In S514, the CPU 301 determines whether all the body parts to be photographed have been photographed. If it determines that they have been photographed, the process proceeds to S515. If it determines that there are body parts that have not been photographed, the process returns to S503. In this embodiment, if, for a particular patient, the frontal view, mandible, maxilla, left view, and right view are stored in the RAM 303 as photographed body part information, the CPU 301 determines that all body parts have been photographed.

[0057] In S515, the CPU 301 determines whether the photographer has turned off the power by pressing the power button 401. If it determines that the power has been turned off, the series of processes is terminated. If the power has not been turned off, the process returns to S501.

[0058] [Image processing based on supplementary information] Figure 6 is a flowchart showing image processing based on supplementary information in this embodiment. Here, we will describe how the image processing apparatus 101 performs horizontal inversion and vertical rotation processing using supplementary information associated with image data.

[0059] In S601, the CPU 201 reads the image file received from the imaging device 108 into the RAM 203. Then, in S602, the CPU 201 reads supplementary information from within the image file. This supplementary information includes the selected area information and mirror usage information associated with each individual image file.

[0060] In S603, the CPU201 uses mirror usage information to determine whether the image data stored in the image file was taken using a mirror. If it is determined that a mirror was used, the process proceeds to S604. If it is determined that a mirror was not used, the process proceeds to S606.

[0061] In S604, the CPU201 reverses the left-right orientation of the image data within the image file. This is because, when using a mirror, the left-right orientation of the image is reversed compared to the image seen when actually observing the subject. In S605, the CPU201 records left-right reversal information associated with the image, indicating that the image has been reversed. In this embodiment, this information is recorded as header information for the image file.

[0062] In S606, the CPU201 determines whether the selected area information is the mandible or not. If it is the mandible, the process proceeds to S607; otherwise, it proceeds to S609. As mentioned above, when the occlusal surface is photographed using a mirror with the patient lying down, the dental arch generally appears convex in both the maxilla and mandible. Therefore, in the case of the mandible, in S607, the CPU201 rotates the image data in the image file by 180 degrees so that the dental arch appears concave. Subsequently, in S608, the CPU201 records 180-degree rotation information, indicating that the image data has been rotated by 180 degrees, in association with the image data. In this embodiment, this information is recorded as header information for the image file.

[0063] In S609, the CPU 201 arranges and displays five images on the display 204 according to the selected area information. An example of the user interface at this time is explained with reference to Figure 7. Once the image display is complete, the process is terminated.

[0064] [User interface for image processing equipment] Figure 7 shows an example of the user interface of the image processing apparatus according to this embodiment. Referring to Figure 7, a method for arranging and displaying image data according to selected area information will be explained.

[0065] 700 is the exit button; when pressed by the user, CPU201 determines that the user has given an exit command and terminates the series of processes. Area 701 is the maxillary occlusal view display area. CPU 201 displays image data in this area for which the selected area information of the image file is the maxilla. 702 is the mandibular occlusal view display area. CPU201 displays image data in this area for which the selected area information of the image file is the mandible.

[0066] 703 is the front view display area. The CPU 201 displays image data in this area if the selected area information of the image file is the front view. 704 is the left-side view display area. The CPU 201 displays image data in this area for which the selected area information of the image file is the left side. 705 is the right-side view display area. The CPU 201 displays image data in this area for which the selected area information of the image file is the right side.

[0067] Areas 706, 707, 708, 709, and 710 are selected area information display areas, and the CPU 201 displays the selected area corresponding to each image. If the selected area information set by the imaging device 108 is incorrect, clicking this area with an input device such as a mouse may display a pop-up menu, allowing the user to change the information.

[0068] Areas 711, 712, 713, 714, and 715 are mirror usage information display areas. The CPU 201 displays the mirror usage information corresponding to each image. If the mirror usage information set by the imaging device 108 is incorrect, clicking this area with an input device such as a mouse may display a pop-up menu, allowing the user to change the information.

[0069] Icons 716, 717, 718, and 719 are horizontally flipped icons. CPU201 displays icons 716, 717, 718, and 719 only if there is horizontally flipped information corresponding to each image.

[0070] Icon 720 represents a 180-degree rotation. CPU201 displays icon 720 only if there is corresponding 180-degree rotation information for each image.

[0071] 721 is a mark indicating the presence of residual teeth. If there is information that estimates the condition of the teeth by analyzing the image, displaying it superimposed on the image will be an aid in diagnosis. For information related to the coordinate position of the image as described above, the coordinate position should be flipped horizontally and / or rotated 180 degrees along with the horizontal flipping and / or rotation.

[0072] 722 is a patient information display area, which displays the patient's name, gender, and age at the time of examination. In this embodiment, images displayed in display areas 701 to 705 are shown only if they are of the same patient and the examination dates match. If these conditions are not met, the images are not displayed on the same screen.

[0073] 723 is the inspection date and time display area. In this embodiment, it is assumed that the correct date and time are set in the imaging device, and that the shooting date and time and the inspection date and time coincide.

[0074] Button 724 is the update button. If the user makes a change in the selected area information display area or the mirror usage information display area, the CPU 201 will re-execute the series of processes shown in Figure 6 according to the updated information, in response to the press of the update button 724, as if the user had performed a display update operation. Also, if the original image display button 725 was displaying the image before horizontal inversion or rotation, pressing the update button 724 will also re-execute the series of processes shown in Figure 6.

[0075] When the user presses the original image display button 725, the CPU 201 recognizes that the user has requested the display of the original image and displays the image immediately after shooting, before horizontal flipping or rotation, in the respective image area. Alternatively, when the original image display button 725 is pressed, the system may switch to displaying the edited image. Switching to displaying the edited image makes it easy to revert from the image immediately after shooting to the edited image.

[0076] In this embodiment, S604 and S607 update the image data in the image file in accordance with horizontal inversion and 180-degree rotation. However, the present invention is not limited to this, and the image data itself in the image file may not be edited, but only the horizontal inversion flag and 180-degree rotation flag may be associated with the image and recorded. In that case, when the user interface shown in Figure 7 is displayed, the image data read into RAM 203 can be horizontally inverted or rotated according to the flag. By doing so, the processing time required to write to the image file related to horizontal inversion and 180-degree rotation can be shortened. In that case, the horizontal inversion icons 716, 717, 718, 719 and the 180-degree rotation icon 720 will display the corresponding icon depending on whether horizontal inversion or rotation is performed at the time of display. Furthermore, when the original image display button 725 is pressed, horizontal inversion or rotation processing according to the flag will not be performed when displaying the image data.

[0077] As described above, according to the first embodiment, the image data is flipped horizontally or rotated 180 degrees based on the selected area information and mirror usage information selected during imaging. This allows the image to be displayed or recorded in a position that matches how it appears when actually observing the patient, enabling medical professionals to make diagnoses using appropriate images.

[0078] Furthermore, by displaying images associated with the patient and the same examination date as a pair, it is possible to prevent diagnostic and recording errors caused by discrepancies between the patient and the images.

[0079] Furthermore, by reviewing the intraoral photographs while checking the selected area information and mirror usage information selected during imaging, any errors can be corrected. Then, by updating the display based on the corrected information, any errors made during imaging can be corrected and recorded.

[0080] Furthermore, by performing horizontal flipping and rotation along with the image, the information recorded in association with the image coordinates can be appropriately utilized by medical professionals to aid in diagnosis.

[0081] Furthermore, by recognizing whether an image is unedited or has been edited (e.g., flipped or rotated), if you need to check the image immediately after shooting, you can view the image data before flipping or rotating by pressing the "Show Original Image" button.

[0082] Furthermore, by retaining information on mirror usage during imaging of each area, medical institutions that perform intraoral imaging using the same procedure each time eliminate the need to adjust settings each time an image is taken.

[0083] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, when the image data captured and stored for each patient does not have location information and mirror usage information associated and stored, the image processing device 101 will infer the dental formula number, tooth type, tooth condition, and location, and perform left-right inversion and up-down rotation. Note that the device configuration of the dental system in the second embodiment is the same as that described with reference to Figures 1 to 3 in the first embodiment, so the description will be omitted.

[0084] Furthermore, in this embodiment, machine learning is used to infer the tooth formula number, tooth type, tooth condition, and location. Alternatively, any image processing method that yields similar results may be used, such as pattern matching, edge detection, or discrimination using grayscale or color information. Furthermore, the general classification of body parts in this embodiment is as follows: front view, occlusal view, lateral view, and others. In this embodiment, images stored for each patient and stored in the HDD 205 of the image processing device 101 are processed. Images transmitted from external devices, such as images captured by the imaging device 108, are also stored in the HDD 205.

[0085] [Image processing based on inference results] Figure 8 is a flowchart showing the image region estimation process and image processing in this embodiment.

[0086] First, in S800, CPU201 reads the five image files, each stored for each patient using the five-image method, from HDD205 to RAM203. Next, in S801, CPU201 infers whether each of the five image files read in S800 is a frontal, occlusal, lateral, or other view.

[0087] In S802, the CPU201 classifies the five images according to their inference results. If the images were properly captured using the five-image method, one image will be classified as a frontal view, two as occlusal views, and two as lateral views. In S803, if the results do not conform to the five-image method, the CPU201 determines that the input images were not appropriate, displays a warning in S804, and interrupts the subsequent processing. If the results conform to the five-image method, the process proceeds to S805.

[0088] In S805, the CPU201 records the "frontal view" as part of the image information classified as frontal, associating it with the image data, and also infers the dental formula number, tooth type, and condition.

[0089] In S806, the CPU201 determines whether the image is oriented correctly vertically. This is determined by whether the upper teeth are located in the upper part of the image and the lower teeth are located in the lower part of the image. If the image is oriented correctly vertically, the process proceeds to S808. If the image is oriented incorrectly vertically, the process proceeds to S807. For example, when taking an image with a patient lying down, if the image is taken facing the patient's head, the recorded image will be inverted vertically.

[0090] In S807, CPU201 rotates the image data in RAM203, which S806 determined to be oriented incorrectly, by 180 degrees. Then, it stores a 180-degree rotation flag associated with the image data.

[0091] Next, in S808, the CPU201 determines whether or not there are traces of inversion in each of the two images classified as occlusal surfaces. The presence or absence of inversion traces is determined by whether or not the date and time of capture recorded in the header of the image file matches the date and time the file was modified. If they do not match, it is determined that there are traces of inversion. If they match, it is determined that there are no traces of inversion. If there are traces of inversion, the process proceeds to S809. If there are no traces of inversion, the process proceeds to S810.

[0092] In the S809, the CPU201 reverses the left and right orientation of the image data in RAM203 that corresponds to the image without any trace of occlusal surface inversion among the two images classified as occlusal surfaces. Then, it stores a left / right inversion flag associated with the image data. As mentioned above, this is because images of the occlusal surface are generally taken using a mirror.

[0093] In S810, the CPU201 infers the tooth number and tooth condition for two images classified as occlusal surfaces. In this embodiment, the tooth number and tooth condition are inferred using object detection in machine learning, and the inference result is obtained as rectangular information corresponding to coordinates on the image. As a result of the inference, the CPU201 records the location information associated with the image, with the image showing the maxillary teeth being recorded as the maxillary occlusal view image and the image showing the mandibular teeth being recorded as the mandibular occlusal view image. In addition, since a mirror is always used for the occlusal view image, mirror usage information is also recorded in association with the image.

[0094] In S811, CPU201 determines whether the dental arch is convex or concave, even though the mandibular teeth have appeared in S810. The determination of whether the dental arch is convex or concave is made based on the arrangement of the teeth in the image of the rectangle detected. If it is determined that the dental arch is convex despite being in the mandible, the process proceeds to S812. If it is determined that the dental arch is concave in the mandible, the process proceeds to S813.

[0095] In S812, the CPU201 rotates the image data in RAM203, which shows a convex dental arch despite being a mandible on the occlusal surface, by 180 degrees. Then, it stores a 180-degree rotation flag associated with the image data.

[0096] Next, in S813, the CPU201 infers the dental formula number and tooth condition for each of the two images classified as lateral.

[0097] Then, in S814, the CPU201 assumes that the two images classified as lateral views were not taken using a mirror, and compares the inference results for the presence or absence of teeth and metal prostheses, etc., with the inference results for the occlusal surface. At this time, the area of ​​the image where the molars are on the right side and the anterior teeth are on the left side is designated as the left surface, and the area of ​​the image where the molars are on the left side and the anterior teeth are on the right side is designated as the right surface, and is compared with the inference results for the occlusal surface. The comparison results for each tooth, assuming that the images were not taken using a mirror, are then stored in the RAM203.

[0098] Next, in S815, the CPU201 reverses the left and right tooth numbers to compare the inference result of the tooth state assuming that the two images classified as lateral views were taken using a mirror, with the inference result of the occlusal surface. Then, the comparison result for each tooth, assuming that the images were taken using a mirror, is stored in the RAM203.

[0099] In S816, the CPU201 uses the tooth matching results stored in S812 and S813 to determine whether or not each tooth was photographed using a mirror. Details of this determination will be described later with reference to Figure 13.

[0100] In S817, the CPU201 comprehensively evaluates the results for each tooth to determine whether or not the image was taken using a mirror, and then determines whether the image was taken using a mirror, without a mirror, or if it was inconsistent or undecided. If it is inconsistent or undecided, the process proceeds to S820. If it is not inconsistent or undecided, the process proceeds to S818.

[0101] In S818, the CPU 201 determines, based on the overall judgment, which of the two lateral view images is the right or left view. If the image assumed to have been taken using a mirror in S815 is selected, the image data in RAM 203 for the two lateral views is flipped horizontally. A horizontal flip flag is then associated with the image data and stored, along with location information and mirror usage information. If the image assumed to have been taken without using a mirror is selected, the image data for the two lateral views is not flipped horizontally, and location information and mirror usage information are recorded as if no mirror was used.

[0102] In S819, the CPU201 appropriately processes the views classified in the previous steps—front view, maxillary occlusal view, mandibular occlusal view, left lateral view, and right lateral view—as shown in Figure 6, and displays them using the user interface shown in Figure 7. Then, the series of processes is terminated.

[0103] Meanwhile, in S820, CPU201 determines whether there is a contradiction or not. If there is a contradiction, it proceeds to S821. If there is no contradiction, it proceeds to S822.

[0104] In S822, the CPU201 cannot determine whether a mirror is being used or not, so it issues a notification stating, "No characteristic differences were detected between the left and right sides, therefore no correction will be performed." For the two lateral view images, the image with the molars on the right and the front teeth on the left is designated as the left lateral view, and the image with the molars on the left and the front teeth on the right is designated as the right lateral view. It is also assumed that no mirror was used. The system then records the location information and mirror information in association with the image data. After that, it proceeds to S819.

[0105] On the other hand, in S821, CPU201 issues a notification stating, "There may be images from other patients mixed in," because no consistent combination exists, and terminates the series of processes. [Method for inferring the location indicated by the image] Figure 9 is a conceptual diagram showing an example of the classification results of the inferred images. Figure 9(a) shows an image inferred to be the maxilla, and Figure 9(b) shows an image inferred to be the mandible.

[0106] Figures 9(c) and 9(d) show images inferred to be lateral views assuming no mirror is used. Figure 9(c) shows the image inferred to be the right view, and Figure 9(d) shows the image inferred to be the left view. When no mirror is used, the molars appear on the left and the incisors on the right, so the image shown in Figure 9(c) is assumed to be the right view. Similarly, when no mirror is used, the incisors appear on the left and the molars on the right, so the image shown in Figure 9(d) is assumed to be the left view.

[0107] Figures 9(e) and 9(f) show images inferred to be lateral views assuming the use of a mirror. Figure 9(e) is the image inferred to be the right view, and is a horizontally flipped version of the image in Figure 9(d). When using a mirror, the molars appear on the left and the front teeth on the right, so the image shown in Figure 9(e) is assumed to be the right view. Similarly, Figure 9(f) is the image inferred to be the left view, and is a horizontally flipped version of the image in Figure 9(c). When using a mirror, the front teeth appear on the left and the molars on the right, so the image shown in Figure 9(f) is assumed to be the left view.

[0108] Figures 10, 11, and 12 are tables showing the tooth condition, with the inference results for the tooth location corresponding to the dental formula number. The tooth condition is determined by the overlap between the image coordinates representing the inference results and the image coordinates representing the inference results for the tooth corresponding to the dental formula number.

[0109] Figure 10(a) is a table showing an example of the inference results for the state of teeth 901 to 914 in the maxillary image shown in Figure 9(a). In the image of Figure 9(a), tooth 907 corresponds to dental formula number uR1 in Figure 10(a), and tooth 901 corresponds to dental formula number uR7. Also, tooth 908 corresponds to dental formula number uL1, and tooth 914 corresponds to dental formula number uL7. 915 is a characteristic area in the oral cavity called the palatal fold, which is found only in the maxilla.

[0110] Figure 10(b) is a table showing an example of the inference results for the state of teeth 916 to 927 in the mandibular image shown in Figure 9(b). In the image shown in Figure 9(b), tooth 922 corresponds to dental formula number bR1 in Figure 10(b), and tooth 916 corresponds to dental formula number bR7. Tooth 923 corresponds to dental formula number bL1, and tooth 927 corresponds to dental formula number bL5. 928 is the tongue. 929 is the lingual frenulum, a characteristic part found only in the mandible.

[0111] As shown in Figure 10, the tooth formula number and the inference result are stored. In this embodiment, if a tooth corresponding to the tooth formula number cannot be detected from the image by inference, it is treated as missing.

[0112] For example, Figure 10(a) shows that tooth 907 with dental formula number uR1 is healthy, and tooth 906 with dental formula number uR2 is missing. Dental formula numbers uR3 to uR7 correspond to teeth 905 to 901, and dental formula numbers uL1 to uL7 correspond to teeth 908 to 914. Teeth corresponding to dental formula numbers uR8 and uL8 cannot be detected and are therefore treated as missing. The same applies to Figure 10(b).

[0113] Figure 11(a) is a table showing an example of the inference result of the state of teeth 930 to 944 in the right-side image, assuming that a mirror is not used, as shown in Figure 9(c). In the image of Figure 9(c), tooth 936 corresponds to dental formula number uR1 in Figure 11(a), and tooth 930 corresponds to dental formula number uR7. Also, tooth 942 corresponds to dental formula number bR1, and tooth 938 corresponds to dental formula number bR5. Since Figure 9(c) is inferred assuming a right-side image, the resulting dental formula numbers in Figure 11(a) are all teeth that appear on the right side of the patient. That is, tooth 937 corresponds to dental formula number uL1, tooth 944 corresponds to dental formula number bL2, and tooth 943 corresponds to the tooth to the left of dental formula number bL1, so they are not shown in the table in Figure 11(a). In addition, teeth 945 and 946 are teeth that appear outside the dental arch and may rarely appear in the image of the subject when a mirror is used.

[0114] In the example shown in Figure 11(a), for instance, tooth 936 with dental formula number uR1 is shown to be healthy. Note that teeth corresponding to dental formula numbers uR8, bR6, bR7, and bR8 cannot be detected and are therefore treated as missing.

[0115] Figure 11(b) is a table showing an example of the inference result of the state of teeth 947 to 961 in the left-side image, assuming no mirror is used, as shown in Figure 9(d). In the image of Figure 9(d), tooth 947 corresponds to dental formula number uR1 in Figure 11(b), tooth 948 corresponds to dental formula number uL1, and tooth 953 corresponds to dental formula number uL7. Also, tooth 954 corresponds to dental formula number bR1, tooth 955 corresponds to dental formula number bL1, and tooth 961 corresponds to dental formula number bL7. Since Figure 9(d) is inferred assuming a left-side image, the resulting dental formula numbers in Figure 11(b) will only include teeth that appear on the left side of the patient. Note that teeth 962 and 963 are teeth that appear outside the dentition and may rarely appear in the image when a mirror is used.

[0116] In the example shown in Figure 11(b), for instance, tooth 948 with dental formula number uL1 is shown to be healthy. Note that teeth corresponding to dental formula numbers uL2, uL8, and bL8 cannot be detected and are therefore treated as missing.

[0117] Figure 12(a) is a table showing an example of the inference result for the state of teeth 964 to 978 in the right-side image assuming the use of a mirror as shown in Figure 9(e), with the dental formula numbers L (left) replaced with R (right) as in the inference result shown in Figure 11(b). In the image of Figure 9(e), tooth 970 corresponds to dental formula number uL1 in Figure 12(a), tooth 969 corresponds to dental formula number uR1, and tooth 964 corresponds to dental formula number uR7. Also, tooth 971 corresponds to dental formula number bL1, tooth 972 corresponds to dental formula number bR1, and tooth 978 corresponds to dental formula number bR7. Since Figure 9(e) is inferred assuming a right-side image, the resulting dental formula numbers in Figure 12(a) are all teeth that appear on the right side of the patient. Teeth 979 and 980 are teeth that appear outside the dentition and may rarely appear in the image when a mirror is used.

[0118] In the example shown in Figure 12(a), for instance, tooth 969 with dental formula number uR1 is shown to be healthy. Note that teeth corresponding to dental formula numbers uR2, uR8, and bR8 cannot be detected and are therefore treated as missing.

[0119] Figure 12(b) is a table showing an example of the inference result for the state of teeth 981 to 995 in the left-side image assuming the use of a mirror as shown in Figure 9(f), with the R (right) in the dental formula numbers of the inference result shown in Figure 11(a) changed to L (left). In the image of Figure 9(f), tooth 981 corresponds to dental formula number uR1 in Figure 12(b), tooth 982 corresponds to dental formula number uL1, and tooth 988 corresponds to dental formula number uL7. Also, tooth 995 corresponds to dental formula number bR2, tooth 994 corresponds to dental formula number bR1, tooth 993 corresponds to dental formula number bL1, and tooth 989 corresponds to dental formula number bL5. Since Figure 9(f) is inferred assuming a left-side image, the resulting dental formula numbers in Figure 12(b) are all teeth that appear on the left side of the patient. Note that teeth 996 and 997 are teeth that appear outside the dental arch and may occasionally be visible in the image when using a mirror.

[0120] In the example shown in Figure 12(b), for instance, tooth 982 with dental formula number uL1 is shown to be healthy. Note that teeth corresponding to identification numbers uL8, bL6, bL7, and bL8 cannot be detected and are therefore treated as missing.

[0121] [Explanation of Overall Assessment] Next, we will explain how to determine whether a mirror was used for the lateral view, based on the inferences made regarding the occlusal and lateral views described above.

[0122] Figure 13 is a table showing the degree of agreement between the occlusal and lateral tooth conditions and the patterns of results estimated from that condition. Referring to this table, for each tooth with the same dental formula number shown in Figures 10 to 12, it is determined whether the lateral view is the right-hand image or the left-hand image. As shown in Figure 13, the combinations can be divided into 11 patterns in total, regardless of whether they are consistent or inconsistent (indicated by *), and the degree of agreement between the tooth conditions for each dental formula number falls into one of these 11 patterns. The determination method in S816 of Figure 8 will be explained below using Figure 13.

[0123] In Figure 13, "Image 1" and "Image 2" show images of the side view that were captured. When Image 1 is the right side without image inversion, "Assuming Right" in Image 1 corresponds to the image assuming no mirror was used (for example, Figure 9(c)). Similarly, "Assuming Left" in Image 1 corresponds to the image assuming mirror was used (for example, Figure 9(f)). Likewise, when Image 2 is the left side without image inversion, "Assuming Right" in Image 2 corresponds to the image assuming mirror was used (for example, Figure 9(e)), and "Assuming Left" in Image 2 corresponds to the image assuming no mirror was used (for example, Figure 9(d)).

[0124] Then, the tooth condition in Image 1 and Image 2, assuming they are on the "right" side and assuming they are on the "left" side, is compared with the tooth condition in the occlusal view for the same tooth number. For each tooth, it is determined whether Image 1 and Image 2 are estimated to be the right image or the left image.

[0125] ○ indicates that the condition of the tooth with the same formula number in the occlusal view matches that of images 1 and 2, while × indicates a mismatch. ※ indicates that the condition is not related to alignment or mismatch (either alignment or mismatch is acceptable).

[0126] For example, assuming Image 1 is on the right (for example, Figure 9(c)), the estimated tooth condition of tooth formula number uR1 shown in Figure 11(a) is "healthy," and the corresponding occlusal view of tooth formula number uR1 shown in Figure 10(a) is also "healthy," showing a match (〇). Also, assuming Image 1 is on the left (for example, Figure 9(f)), the estimated tooth condition of tooth formula number uL1 shown in Figure 12(b) is "healthy," and the corresponding occlusal view of tooth formula number uL1 shown in Figure 10(a) is also "healthy," showing a match (〇). Also, assuming Image 2 is on the right (for example, Figure 9(e)), the estimated tooth condition of tooth formula number uR1 shown in Figure 12(a) is "healthy," and the corresponding occlusal view of tooth formula number uR1 shown in Figure 10(a) is also "healthy," showing a match (〇). Furthermore, assuming that Image 2 is the left tooth (for example, Figure 9(d)), the estimated result in Figure 11(b) shows that the tooth with formula number uL1 is "healthy," and the corresponding occlusal view in Figure 10(a) also shows that the tooth with formula number uL1 is "healthy," and they match (〇). In this case, it corresponds to pattern 1 in Figure 13, and the determination of whether Image 1 and Image 2 are right or left images remains unresolved.

[0127] Similarly, assuming Image 1 is the right, the estimated tooth condition for tooth formula number uR7 shown in Figure 11(a) is a "silver prosthesis," while the corresponding occlusal view of tooth formula number uR7 shown in Figure 10(a) is a "silver full crown," which is inconsistent (×). On the other hand, assuming Image 1 is the left, the estimated tooth condition for tooth formula number uL7 shown in Figure 11(b) is a "silver prosthesis," and the corresponding occlusal view of tooth formula number uL1 shown in Figure 10(a) is also a "silver prosthesis," which is consistent (〇). Furthermore, assuming Image 2 is the right, the estimated tooth condition for tooth formula number uR7 shown in Figure 12(a) is a "silver full crown," and the corresponding occlusal view of tooth formula number uR7 shown in Figure 10(a) is also a "silver full crown," which is consistent (〇). On the other hand, assuming Image 2 is the left image, the estimated result in Figure 12(b) shows that the tooth with dental formula number uL1 has a "silver prosthesis," and the corresponding occlusal view of the tooth with dental formula number uL1 in Figure 10(a) also shows a "silver prosthesis," which matches. In this case, it corresponds to pattern 4 in Figure 13, and Image 1 is determined to be the right image and Image 2 is the left image.

[0128] In this way, the same determination is made for all teeth using the table in Figure 13, and based on the state of each tooth number in Image 1 and Image 2 and the state of the corresponding tooth in the occlusal view, it is determined whether Image 1 and Image 2 fall under "Undecided," "Right," "Left," or "Contradictory" as shown in the determination column of Figure 13. Then, the obtained determination results are integrated to determine whether Image 1 and Image 2 are right circles or left faces. Specifically, the number of times Image 1 is determined to be right (or left) is compared with the number of times Image 2 is determined to be right (or left), and the image with the larger number of right (or left) determinations is determined to be the right face (or left face), and the other image is determined to be the left face (or right face).

[0129] However, if there are many "undecided" or "contradictory" cases, or if, for example, the number of times image 1 is determined to be right and the number of times it is determined to be left are the same or nearly the same, it is not possible to determine whether it is the right or left side (NO in S817). In such cases, in S820, it is determined whether the determination result is contradictory or whether a determination cannot be made due to many undecided cases. If it is contradictory, proceed to S822; otherwise, proceed to S821.

[0130] In this embodiment, in S805, S805, and S810, the image data in RAM203 is inverted or rotated and stored with a associated flag. However, the image data in the image file may be inverted and rotated and rewritten. The display processing by updating the flag or the image file itself is the same as the method described in the first embodiment.

[0131] As described above, according to this second embodiment, even when part information and mirror usage information are not associated and stored with the image, it is possible to highly estimate the part indicated by the image and perform left-right flipping and up-down rotation processing.

[0132] <Variation> In the second embodiment described above, the state of each tooth was estimated, and the area indicated by the image was estimated by analyzing the estimated results. In contrast, this modified example describes an example of a different estimation method.

[0133] Figure 14 is a conceptual diagram showing the detection results of characteristic non-tooth tissues and instruments. 1401 is the palatal fold, 1402 is the tongue, and 1403 and 1404 are the angular hooks of the mouth.

[0134] In the second embodiment, S810 determines whether it is the maxilla or mandible based on the tooth inference result, but if the palatal fold 1401 is detected, it may be determined to be the maxilla. Alternatively, if the tongue 1402 or the lingual frenulum 929 shown in Figure 9(b) is detected, it may be determined to be the mandible.

[0135] Furthermore, in S813, the state of the teeth in the lateral view image is inferred, and in S816, the overall judgment determines whether the image is a right or left view. However, if there are orthoscopic clasps 1403 and 1404, the mirror usage information may be recorded as not being used. Also, if the date and time the image file was taken and the date and time the file was modified match, the right or left lateral view may be determined based on the position of the anterior and posterior teeth without reversing the image left and right.

[0136] Furthermore, if teeth 945, 946, 979, and 980 appear outside the dentition as shown in Figure 9, the use of a mirror may be recorded as such. If the image file's capture date and time match the file's modification date, the image may be reversed horizontally to determine whether it is a right-sided or left-sided view based on the relative positions of the anterior and posterior teeth.

[0137] Furthermore, in the second embodiment, an example was described in which the dental formula number and the state of the teeth are strictly matched for the occlusal and lateral views. However, given the certainty that the set of images is taken from the oral cavity of a single patient, the matching can be determined by the presence or absence of teeth or by characteristic teeth such as full crowns. Alternatively, the matching can be determined by the type of tooth, such as lateral incisors, canines, or molars, which is less precise than the dental formula number. Even if the accuracy of the inference is not very high, the images can be flipped left to right or rotated 180 degrees and displayed in a way that matches how they appear when observing the patient.

[0138] As described above, according to this modified version, even if location information and mirror usage information are not recorded at the time of imaging, the location indicated by the image can be estimated based on the type and condition of the teeth, the detection results of characteristic locations and instruments in the oral cavity, etc.

[0139] <Other Embodiments> Furthermore, the present invention may be applied to a system consisting of multiple devices (e.g., a host computer, interface devices, scanners, video cameras, etc.) or to a device consisting of a single device (e.g., a copier, a facsimile machine, etc.).

[0140] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0141] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0142] 101: Image processing device, 102: Network, 103: Dental electronic medical record display terminal, 104: Dental electronic medical record system, 105: Dental information DB, 106: In-hospital system, 107: Patient information DB, 108: Imaging device, 201: CPU, 202: ROM, 203: RAM, 204: Display, 206: Input device, 207: I / F

Claims

1. An acquisition means for acquiring an image file including first image data of an intraoral image, site information indicating a site in the oral cavity, and mirror use information indicating whether a mirror was used at the time of photographing; a processing means for determining whether or not at least one of left-right inversion processing and up-down rotation processing is necessary for the first image data of the acquired image file based on the part information and the mirror use information, and for performing the processing determined to be necessary.

13. An image processing device comprising:

2. The image processing device according to claim 1 , characterized in that the site information added to the first image data of the intraoral image is any one of the front, upper jaw, lower jaw, right side, and left side in the five-image method.

3. The image processing apparatus according to claim 2 , wherein the processing means performs the up-down rotation processing on the first image data when the part information is a lower jaw.

4. 4. The image processing device according to claim 1, wherein the processing means performs the left-right inversion processing on the first image data when the mirror use information indicates that a mirror has been used.

5. 5. The image processing apparatus according to claim 1, wherein, when the first image data is processed by the processing means, second image data obtained by the processing is recorded in the image file.

6. 5. The image processing apparatus according to claim 1, wherein, when the first image data is processed by the processing means, a flag indicating the content of the processing is added to the first image data and recorded.

7. The image processing device further includes a display means, An image processing device as described in any one of claims 1 to 6, characterized in that when the first image data of a corresponding part based on the part information, or second image data obtained by processing the first image data by the processing means, is present in a display area of ​​the display means predetermined according to the part, the second image data is displayed as third image data.

8. displaying, on the display means, at least one of the part information and the mirror use information of the first image data corresponding to the third image data in association with the third image data; 8. The image processing apparatus according to claim 7, further comprising an input unit for changing the body part information and the mirror use information displayed on the display unit.

9. a selection means for selecting the first image data before being processed by the processing means; 9. The image processing apparatus according to claim 7, wherein when the first image data is selected by the selection means, the first image data is displayed on the display means.

10. 10. The image processing apparatus according to claim 7, further comprising: displaying, on said display means, information on a patient and a date of imaging associated with said first image data corresponding to said third image data.

11. An imaging means for capturing an intraoral image; A setting means for setting site information indicating a site in the oral cavity and mirror use information indicating whether a mirror was used or not when photographing; a transmission means for transmitting to an external device an image file in which the image data of each intraoral image captured by the imaging means is added with site information and mirror use information that were set when the image was captured; An imaging device comprising:

12. 12. The imaging apparatus according to claim 11, further comprising a storage unit for storing the body part information and the mirror use information set by the setting unit.

13. An imaging system including an imaging device and an image processing device, The imaging device, An imaging means for capturing an intraoral image; A setting means for setting site information indicating a site in the oral cavity and mirror use information indicating whether a mirror was used or not when photographing; A transmitting means for transmitting an image file in which the site information and the mirror use information set at the time of photographing are added to the first image data of each intraoral image photographed by the imaging means to the image processing device, The image processing device, a receiving means for receiving the image file from the imaging device; a processing means for determining whether or not at least one of left-right inversion processing and up-down rotation processing is necessary for the first image data of the received image file based on the part information and the mirror use information, and for performing the processing determined to be necessary.

1. An imaging system comprising:

14. An acquisition means for acquiring image data of a plurality of intraoral images; A determination means for determining whether or not site information indicating a site in the oral cavity and mirror use information indicating whether a mirror was used or not at the time of photographing are added to the image data of each intraoral image acquired by the acquisition means; An estimation means for estimating whether the site and the mirror are used based on features of each of the intraoral images when the site information and the mirror use information are not added; a storage means for adding site information and mirror use information to the image data of each of the intraoral images based on the site estimated by the estimation means and the use or non-use of the mirror; 13. An image processing device comprising:

15. 15. The image processing device according to claim 14, further comprising a processing means for determining whether at least one of left-right inversion processing and up-down rotation processing is necessary for the image data stored in the memory means based on the part information and the mirror use information, and performing the processing determined to be necessary.

16. The estimation means includes: A first estimation of a site indicated by each of the intraoral images among a plurality of predetermined intraoral sites based on a feature of each of the intraoral images; a second estimation of whether or not a mirror was used when at least any of the intraoral images among the plurality of intraoral images was taken based on the state of each tooth in the plurality of intraoral images estimated based on the estimated location; and 16. The image processing apparatus according to claim 14, further comprising:

17. The site is any one of the front, upper jaw, lower jaw, right side, and left side in the five-plate method, The estimation means, in the second estimation, estimating the state of each tooth from the intraoral images estimated to be the upper jaw and the lower jaw, estimating the state of each tooth from the intraoral images estimated to be the right side and the left side when it is assumed that a mirror was not used when photographing, and estimating the state of each tooth from the intraoral images estimated to be the right side and the left side when it is assumed that a mirror was used when photographing; The image processing device described in claim 16, characterized in that it estimates whether a mirror was used when the intraoral images estimated to be the right and left sides were taken based on the consistency between the state of each tooth of the upper and lower jaw and the state of each tooth on the right and left sides assuming that a mirror was not used when taking the images, and the consistency between the state of each tooth of the upper and lower jaw and the state of each tooth on the right and left sides assuming that a mirror was used when taking the images.

18. The image processing device according to claim 14 or 15, wherein the estimation means estimates a region indicated by each of the intraoral images from among a plurality of predetermined intraoral regions based on features of each of the intraoral images.

19. The image processing device according to claim 14 , wherein the estimation means estimates the location of each of the intraoral images based on a detection result of tissue other than teeth in each of the intraoral images.

20. The image processing device according to claim 14 , wherein the estimation means estimates the location of each of the intraoral images based on a detection result of the type of tooth in each of the intraoral images.

21. The image processing device according to any one of claims 14, 15, and 18, characterized in that the estimation means estimates whether or not a mirror was used when each of the intraoral images was taken based on a detection result of a predetermined instrument in each of the intraoral images.

22. The image processing device according to any one of claims 14, 15, and 18, characterized in that the estimation means estimates whether or not a mirror was used when each of the intraoral images was taken based on a detection result of whether or not each of the intraoral images contains reflections of teeth outside the dentition.

23. An acquisition step in which an acquisition means acquires an image file including image data of an intraoral image, site information indicating a site in the oral cavity, and mirror use information indicating whether a mirror was used at the time of photographing; a processing step in which a processing means judges whether or not at least one of left-right inversion processing and up-down rotation processing is necessary for the image data of the image file based on the part information and the mirror use information, and performs the processing that is judged to be necessary; 13. An image processing method comprising:

24. An imaging step in which an imaging means captures an intraoral image; A setting step in which a setting means sets site information indicating a site in the oral cavity and mirror use information indicating whether or not a mirror is used when photographing; A transmission step in which a transmission means transmits to an external device an image file in which the image data of each intraoral image captured by the imaging means is added with site information and mirror use information that were set when the image was captured.

13. An imaging method comprising:

25. In the imaging device, An imaging step of capturing an intraoral image; A setting step of setting site information indicating a site in the oral cavity and mirror use information indicating whether or not a mirror was used when photographing; A generating step of generating an image file in which the site information and mirror use information set at the time of photographing are added to the image data of each of the photographed intraoral images. In the image processing device, a processing step of determining whether or not at least one of left-right inversion processing and up-down rotation processing is necessary for the image data of the image file generated in the generating step, based on the part information and the mirror use information, and performing the processing determined to be necessary.

13. An image processing method comprising:

26. An acquisition step in which an acquisition means acquires image data of a plurality of intraoral images; A judgment step in which the judgment means judges whether or not site information indicating a site in the oral cavity and mirror use information indicating whether a mirror was used at the time of photographing are added to the image data of each acquired intraoral image. An estimation step in which, when the site information and the mirror use information are not added, an estimation means estimates the site and the presence or absence of use of the mirror based on features of each of the intraoral images; a storage step of storing the site information and mirror use information in a storage medium by a storage means, the site information and mirror use information being added to the image data of each intraoral image based on the site estimated in the estimation step and the presence or absence of the use of the mirror; 13. An image processing method comprising:

27. A program for causing a computer to execute each step of the method according to any one of claims 23 to 26.

28. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 10 and 14 to 22.

29. A computer-readable storage medium storing the program according to claim 27 or 28.

Images