Image processing device, imaging system, and image processing method

Abstract

To make a plurality of captured intraoral images look the same as an appearance when a patient is observed.SOLUTION: In an imaging device, intraoral images are captured, and portion information indicating a portion in the oral cavity and mirror use information indicating presence or absence of use of a mirror in capturing the images are set. An image file is generated by adding the portion information and the mirror use information set in capturing the images to image data on the respective captured intraoral images. An image processing device determines whether or not it is necessary to execute at least one of right-left inversion processing and vertical rotation processing on the basis of the portion information and the mirror use information for the image data on the image file generated by the imaging device, and executes the processing that it determines as necessary.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing device. , taken Image system, and Image Processing The present invention relates to a method, and in particular to a technique for processing captured intraoral images. [Background technology]

[0002] In oral surgery and dentistry, medical professionals take intraoral photographs for the purpose of diagnosing and recording the oral cavity. Using these intraoral photographs, it is possible to formulate treatment strategies and plans, as well as evaluate the post-treatment process. There are various methods for taking photographs of the patient's oral cavity from multiple angles, such as the 5-photo method, the 9-photo method, and the 12-photo method. Of these, the 5-photo method is widely used as it allows recording the condition of the oral cavity with a small number of photographs. Visible light intraoral images have the advantage of being able to diagnose and record the condition of the gums, tooth color, etc.

[0003] Meanwhile, radiographs are also used as another method for diagnosing and recording the oral cavity. Patent Document 1 discloses an image processing device that has an image size adjustment unit and an image brightness adjustment unit, and adjusts the brightness to match the average of a histogram in order to make the size and brightness uniform when stitching together multiple radiographs taken from multiple directions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-67532 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the technology disclosed in Patent Document 1 can match the brightness and size to X-ray photographs, it cannot match the appearance when observing a patient with intraoral photographs, which are taken in different orders or through a mirror.

[0006] The present invention has been made in consideration of the above problems, and aims to make the appearance of multiple captured intraoral images the same as how they appear when observing a patient. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the image processing device of the present invention has an acquisition means for acquiring an image file including first image data of an intraoral image, area information indicating an area within the oral cavity, and mirror use information indicating whether a mirror was used or not when the image was taken, and a processing means for determining whether at least one of left-right flip processing and up-down rotation processing is necessary for the first image data of the acquired image file based on the area information and the mirror use information, and performing the processing determined to be necessary. [Effects of the Invention]

[0008] According to the present invention, it is possible to make the appearance of a plurality of captured intraoral images the same as how they appear when observing a patient. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the device configuration of a dental system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram showing the hardware configuration of an image processing apparatus according to an embodiment. [Figure 3] 1 is a block diagram showing a schematic functional configuration of a digital still camera according to an embodiment. [Figure 4] FIG. 1 is a schematic rear view of a digital still camera according to an embodiment. [Figure 5] 5 is a flowchart showing an imaging control process in the first embodiment. [Figure 6] 5 is a flowchart showing image processing based on supplementary information in the first embodiment. [Figure 7] FIG. 2 is a view showing an example of a user interface of the image processing apparatus according to the first embodiment. [Figure 8] 10 is a flowchart showing the process of estimating a region of an image and the image processing in the second embodiment. [Figure 9] 10A and 10B are conceptual diagrams showing possible arrangements of captured images in the second embodiment. [Figure 10] FIG. 10 is a diagram showing a table of inference results of the state of each tooth based on an occlusal view image in the second embodiment. [Figure 11] FIG. 10 is a table showing the inference results of the state of each tooth based on a side view image, assuming that a mirror is not used, in the second embodiment. [Figure 12] FIG. 10 is a table showing the inference results of the state of each tooth based on a side view image, assuming the use of a mirror, in the second embodiment. [Figure 13] FIG. 10 is a table showing inferred patterns of image arrangement based on the state of each tooth in the second embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a captured image in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] First Embodiment [Definition of words] In this embodiment, the intraoral photographs are image data of visible light captured using a five-photo method used in dental diagnosis. The five photos in the five-photo method are a front view, an occlusal view of the upper jaw, an occlusal view of the lower jaw, a left lateral view, and a right lateral view. In this embodiment, the occlusal views of the upper and lower jaws are collectively referred to as the occlusal view. The left and right lateral views are collectively referred to as the lateral view. Furthermore, the intraoral region in this embodiment refers to any of the front, upper jaw, lower jaw, left side, and right side.

[0012] In this embodiment, the dental formula number is a symbol used to represent the position of a tooth in a dental formula, and is assigned in ascending order from the front to the back in each of the top, bottom, left, and right directions. In the case of permanent teeth, the central incisor is number 1 and the third molar is number 8. In the case of primary teeth, the primary central incisor is number A and the second primary molar is number E. In this invention, the dental formula number refers to a name such as "upper right number 6." In contrast, the tooth type is a name based on the shape of the tooth. In order from the front in each of the top, bottom, left, and right directions, the teeth are: central incisor, lateral incisor, canine, (first and second) premolar, (first, second, third) molar. In the case of primary teeth, the teeth are: primary central incisor, primary lateral incisor, primary canine, (first and second) primary molar. The dental formula number identifies a single tooth and is described separately from the tooth type.

[0013] In this embodiment, the first to eighth teeth located on the right side of the upper jaw are represented as uR1 to uR8, and the first to eighth teeth located on the left side of the upper jaw are represented as uL1 to uL8. The first to eighth teeth located on the right side of the lower jaw are represented as bR1 to bR8, and the first to eighth teeth located on the left side of the lower jaw are represented as bL1 to bL8.

[0014] In this embodiment, the tooth condition indicates the health state of the tooth, whether or not dental treatment has been performed, etc. For each tooth, at least one of healthy tooth, caries, filling, crown, implant, etc. is selected. In this embodiment, dental information is information that associates the dental formula number with the condition and image data of each tooth for each patient.

[0015] [Prerequisites for intraoral imaging] For intraoral images taken using the five-photo method, the state in which they appear to be consistent with how they appear when observing a patient is when, for the maxillary occlusal view, the dental arch is convex. For the mandibular occlusal view, the dental arch is concave. For the frontal view, the patient's nose is facing up and their jaw is facing down. For the left lateral view, the patient's nose is facing up and their jaw is facing down, just like the frontal view, with the molars on the right and the front teeth on the left. For the right lateral view, the patient's nose is facing up and their jaw is facing down, just like the frontal view, with the molars on the left and the front teeth on the right.

[0016] The general method for taking intraoral images using the five-photo method is to face the patient directly without using a mirror when taking frontal images. Photographs of the occlusal surface are usually taken using a mirror. This is because it is not possible to photograph the back teeth without using a mirror. When photographing the occlusal surface using a mirror with the patient lying down, the dental arches of both the upper and lower jaws are generally convex. When photographing the lateral surface, a mirror may be used, or the mouth may be opened using a lip clamp without using a mirror. When photographing using a mirror, the left and right sides will be reversed compared to how the patient's oral cavity appears when observed without a mirror.

[0017] It is assumed that for the same patient, the use of mirrors will not change between the left and right sides, such as using a mirror only for the right lateral view and not using a mirror for the left lateral view when taking a series of images.

[0018] [System Configuration] FIG. 1 is a block diagram showing the device configuration of a dental system as an imaging system of this embodiment.

[0019] The imaging device 108 captures an image of the oral cavity 109 of a patient to be examined and generates visible light image data. In the following description, the image captured by the imaging device 108 will be described as visible light image data. Furthermore, patient information such as a patient ID is added to the captured image data and transmitted to the image processing device 101, and the captured image is deleted in response to a request from the image processing device 101. Furthermore, the imaging device 108 is assumed to be equipped with an input / output device (not shown) for checking the captured image and selecting patient information, similar to a commercially available digital camera.

[0020] The image processing device 101 determines the dental almanac number and the state of the teeth based on information including image data transmitted from the imaging device 108. Note that the process of adding information about the intraoral region (front, upper jaw, lower jaw, left side, right side) of the oral cavity image performed by the dental electronic medical record display terminal 103 described later may be performed by the image processing device 101 using a method for determining the dental almanac number and the state of the teeth.

[0021] Furthermore, the image processing device 101 communicates with the imaging device 108 and the dental electronic medical record system 104. When the imaging device 108 is turned on or the patient information is reconfirmed, the image processing device 101 acquires the patient information of the patient being examined from the dental electronic medical record system 104 and transmits it to the imaging device 108.

[0022] The dental electronic medical record display terminal 103 communicates with the dental electronic medical record system 104, displays the dental electronic medical record, accepts input when creating it, and adds information about the intraoral area of ​​the oral cavity image (front, upper jaw, lower jaw, left side, right side).

[0023] The dental electronic medical record system 104 receives a request from the dental electronic medical record display terminal 103, communicates with the image processing device 101, the in-clinic system 106, and the dental information DB 105, and transmits and receives the data required for the dental electronic medical record creation process. The dental information DB 105 communicates with the dental electronic chart system 104, and stores, transmits and receives dental information of patients in association with patient information such as patient IDs. The in-hospital system 106 communicates with the patient information DB 107 to register and acquire patient information, and also communicates with the dental electronic medical record system 104 to transmit the patient information. The patient information DB 107 communicates with the in-hospital system 106 to store, transmit, and receive the patient information. As a result, the dental electronic medical record system 104 acquires information about the patient to be examined from the patient information DB 107 via the in-hospital system 106.

[0024] The image processing device 101, the imaging device 108, the dental electronic medical record system 104, and the in-hospital system 106 communicate with each other via a network 102. Furthermore, communication between the dental electronic medical record display terminal 103 and the dental electronic medical record system 104 is performed via an HDMI (registered trademark) cable or the like. Communication between the dental information DB 105 and the dental electronic medical record system 104, and communication between the in-hospital system 106 and the patient information DB 107 are performed via a USB cable or the like. Note that in the above example, the communication means are a network, HDMI, and USB, but the present invention is not limited to these and may be performed, for example, by wireless communication.

[0025] [Computer operating as an image processing device] 2 is a block diagram showing the hardware configuration of an image processing device 101 according to this embodiment. The image processing device 101 includes a processor (CPU) 201 that executes a program and a ROM (Read Only Memory) 202 that stores the program. It also includes a RAM (Random Access Memory) 203 into which data required for executing the program is loaded. It also includes an HDD (Hard Disk Drive) 205 that stores inference data, inference results, data for generating inference data (e.g., learning data), and the like. It also includes an input device 206 for input used when registering setting information for the program, a display 204 for display confirmation, an I / F (Interface) 207 used for communication with an external system, and a bus 208.

[0026] Each function of the image processing device 101 is realized by loading a predetermined program onto hardware such as a CPU 201 and a ROM 202, and by the CPU 201 performing calculations. Each function is also realized by communication with an external system via an I / F 207, and by reading and writing data from and to a RAM 203 and an HDD 205, and by the CPU 201 performing calculations.

[0027] [Digital still camera operating as an imaging device] 3 is a block diagram showing the functional configuration of a digital still camera as an example of the imaging device 108 in this embodiment. By executing a predetermined control program in the digital still camera, the imaging process described below is realized and the digital still camera functions as the imaging device 108.

[0028] The imaging unit 300 converts an incident optical image into an electrical signal using a solid-state imaging device, and performs analog-to-digital conversion on the obtained electrical signal to generate image data.

[0029] The CPU 301 controls the entire digital still camera. The ROM 302 stores the operation processing procedures of the CPU 301 (for example, programs for processing when the digital still camera is turned on and basic input / output processing). The RAM 303 functions as the main memory of the CPU 301, and various programs including control programs for realizing the processing described below are loaded from the ROM 302 and executed by the CPU 301. The RAM 303 also provides a work area for the CPU 301 when it executes various processes.

[0030] The display device 304 displays various information under the control of the CPU 301. For example, it displays data stored in a storage medium (not shown). The input device 305 is composed of one or a combination of switches, dials, touch panels, pointing devices using line of sight detection, voice recognition devices, etc. for performing various operations, and includes, for example, a release button located on the top of a digital still camera.

[0031] The media drive 306 is configured to accept a removable storage medium and allows data to be stored in the storage medium and data to be read from the storage medium. The network interface 307 is connected to the network 102 via a communication line 309. Through this network interface 307, data is sent and received to and from the image processing apparatus 101, and a server computer and a personal computer (not shown). The system bus 308 comprises an address bus, a data bus, and a control bus, and connects the above-mentioned units.

[0032] The image processing unit 311 performs image processing on the image data output from the imaging unit 300. The CPU 301 temporarily stores the image data generated by the imaging unit 300 and the attribute information at that time in the RAM 303. Then, as necessary, the image processing unit 311 performs a series of image processing to generate image data that matches the visual characteristics of humans.

[0033] Under the control of the CPU 301, the file generation unit 312 converts the image data processed by the image processing unit 311 and stored in the RAM 303 into image data in a general-purpose still image format such as a JPEG image.

[0034] [User interface of imaging device] FIG. 4 is a schematic rear view of a digital still camera that operates as the imaging device 108 of this embodiment.

[0035] 4, power button 401 switches the power on and off. When the photographer presses this button while the digital still camera is off, CPU 301 determines that the photographer has instructed the camera to turn on the power, and turns the power on. When the photographer presses this button while the camera is on, CPU 301 determines that the photographer has instructed the camera to turn off the power, and turns the power off.

[0036] When the power is turned on, display begins on the display 408. Note that Fig. 4 shows an example of a screen for selecting a region of a patient to be photographed in this embodiment (hereinafter referred to as "region selection screen").

[0037] When the photographer presses the release button 402, the CPU 301 determines that an instruction to capture a still image has been issued.

[0038] Reference numeral 403 denotes an up button, 404 a right button, 405 a down button, 406 a left button, and 407 a confirm button, which constitute the input device 305. When the photographer presses any of the direction buttons 403 to 406, the CPU 301 determines that the photographer has performed an operation to switch the selection object, and switches the selection object on the display 408 according to the direction of the pressed direction button 403 to 406. When the photographer presses the confirm button 407, the CPU 301 determines that the photographer has performed an operation to confirm, and stores the selected information in the RAM 303, and switches the state of the digital still camera.

[0039] The display 408 constitutes the display device 304, and together with the touch panel laid on its surface, constitutes the input device 305. When the photographer touches any point on the screen with his / her finger, the CPU 301 determines that an input instruction has been given by the photographer, determines the content of the operation from the content displayed at the touched position, and performs various processes according to the content of the operation.

[0040] Reference numerals 409 to 422 indicate the contents displayed on the display 408, and here they show the contents when the region selection screen is displayed as described above. Region 409 displays a character string that prompts the photographer to select one of the options. Focus frame 410 is a frame for informing the photographer of the item currently being selected. Selection item display region 411 indicates an area that lists the options, and here displays front 413, upper jaw 414, lower jaw 415, left side 416, and right side 417, which are the regions that are candidates when taking an intraoral photograph. Scroll bar 412 allows the selection to be made in selection item display region 411. LimbsWhen all the candidates cannot be displayed, the display area is changed.

[0041] Photography When the photographer presses one of the direction buttons 403 to 406, the CPU 301 determines that the photographer has performed a selection change operation, and moves the focus frame 410 within the selection item display area 411. Alternatively, the photographer can also move the focus frame 410 by touching the area of ​​the desired selection item displayed on the display 408.

[0042] Menus 418, 419, 420, 421, and 422 are associated with whether or not a mirror is to be used when photographing each body part, and are hereinafter referred to as "mirror use information setting menus." The initial value is unset, and when one of the mirror use information setting menus 418, 419, 420, 421, and 422 is selected using focus frame 410, a pull-down menu, for example, is displayed, allowing the user to select whether or not to use a mirror. Alternatively, for example, with one of the mirror use information setting menus 418, 419, 420, 421, and 422 selected using focus frame 410, pressing enter button 407 may cycle through the options; the selection method is not limited.

[0043] In the example shown in Fig. 4, mirror use information setting menu 418 indicates that a mirror is not used when photographing the front. Mirror use information setting menu 419 indicates that a mirror is used when photographing the upper jaw, and mirror use information setting menu 420 indicates that a mirror is used when photographing the lower jaw. Mirror use information setting menus 421 and 422 indicate that whether or not a mirror is used when photographing the left and right sides has not been set.

[0044] When the selection is confirmed by pressing the decision button 407 while the focus frame 410 is positioned at one of the mirror use information setting menus 418 to 422, or by moving to another mirror use information setting menu 418 to 422, the CPU 301 determines the option selected at the time of confirmation.

[0045] Information on whether or not to use a mirror may be stored as mirror use information in association with information on each body part and stored in a non-volatile recording medium via RAM 303 and media drive 306. The purpose of recording mirror use information in a non-volatile recording medium is that if the body parts to use a mirror are consistent at the medical institution, it becomes unnecessary to set the information each time the power is turned on and off. However, it is not necessary to record mirror use information, and it may be configured to be set each time imaging is performed.

[0046] [Image capture control processing] 5 is a flowchart showing the imaging control process in this embodiment. Here, a method for recording body part information and mirror use information in the imaging device 108 in association with image data will be described.

[0047] In S501, the CPU 301 acquires patient information from the image processing device 101. In this embodiment, the patient information includes the patient's name, sex, and age at the time of examination.

[0048] In S502, the CPU 301 displays the patient information received in S501 and prompts the user to confirm whether the information matches the patient being examined. If it is determined that the patient information does not match, the process returns to S501 and the user acquires the patient information again from the image processing device 101. If it is determined that the patient information matches, the process proceeds to S503. The patient information determined to match is stored in the RAM 303.

[0049] In S503, CPU 301 displays the region selection screen shown in Fig. 4. In S504, CPU 301 determines whether a region has been selected. Whether a region has been selected is determined by whether or not the enter button 407 has been pressed while the focus frame 410 is on one of regions 413 to 417. If it is determined that a region has been selected, CPU 301 stores the selected region information in RAM 303 as selected region information, and proceeds to S505. At this time, mirror use information associated with the selected region information is also stored in RAM 303. If it is not determined that a region has been selected, the screen of S503 waits for input from the photographer.

[0050] In S505, the CPU 301 determines whether the mirror use information stored in the RAM 303 in S504 has been set. If the mirror use information setting menu has been set to use or not use the mirror, it determines that the mirror use information has been set, and the process proceeds to S507. If the mirror use information setting menu has not been set, it determines that the mirror use information has not been set, and the process proceeds to S506. In S506, the CPU 301 displays a warning on the display 408, such as "Please set whether to use or not use the mirror," and returns to S503.

[0051] In S507, the CPU 301 transitions to a state (image capture mode) in which the image capture unit 300 can capture an image. In S508, the CPU 301 determines whether or not the release button 402 has been pressed. If the release button 402 has been pressed, the CPU 301 determines that an image capture instruction has been received from the photographer, and proceeds to S509. If the release button 402 has not been pressed, the CPU 301 determines that an image capture instruction has not yet been received from the photographer, and returns to S507.

[0052] In S509, the CPU 301 controls the imaging unit 300 to perform imaging and acquire image data.

[0053] In S510, the CPU 301 controls the image processing unit 311 and the file generation unit 312 to SThe image data obtained in 509 is converted into image data in a general-purpose file format. Then, the patient information, imaging date and time information, selected body part information, and mirror use information stored in RAM 303 are associated with the image data and recorded as supplementary information. In this embodiment, JPEG is used as an example of a general-purpose file format, and the supplementary information is recorded as header information of the JPEG file.

[0054] In S511, CPU 301 displays the image captured in S509 and an OK button on display 408, and asks the photographer whether or not the desired photo was taken. In S512, CPU 301 determines whether or not the OK button has been pressed by the photographer. The OK button has been pressed by a touch operation on display 408 or by pressing or not the decision button 407. If it is determined that the OK button has been pressed, the process proceeds to S513. On the other hand, if it is determined that the OK button has not been pressed for a predetermined time, the process returns to S507, and the photographer is prompted to take the photo again. Note that an NG button may also be displayed on display 408, and if the NG button is pressed, the process may return to S507.

[0055] In S513, the CPU 301 transmits the image file generated in S510 to the image processing device 101 via the network 102. Also, the selected region information is stored in the RAM 303 as imaged region information in association with the patient information.

[0056] In S514, CPU 301 determines whether all regions to be imaged have been imaged. If it is determined that they have been imaged, the process proceeds to S515. If it is determined that there are regions that have not yet been imaged, the process returns to S503. In this embodiment, if the front, lower jaw, upper jaw, left side, and right side are stored in RAM 303 as imaged region information for a specific patient, CPU 301 determines that all regions have been imaged.

[0057] In S515, the CPU 301 determines whether or not the photographer has performed a power-off operation by pressing the power button 401. If it is determined that a power-off operation has been performed, the series of processes ends. If a power-off operation has not been performed, the process returns to S501.

[0058] [Image processing based on additional information] 6 is a flowchart showing image processing based on supplementary information in this embodiment. Here, a method for performing left-right flip processing and up-down rotation processing in the image processing device 101 using supplementary information associated with image data will be described.

[0059] In S601, the CPU 201 loads an image file received from the imaging device 108 into the RAM 203. Then, in S602, the CPU 201 loads additional information from the image file. The additional information in this case is selected body part information and mirror use information associated with each image file.

[0060] In S603, CPU 201 uses the mirror use information to determine whether the image data saved in the image file was captured using a mirror. If it is determined that a mirror was used, the process proceeds to S604. If it is determined that a mirror was not used, the process proceeds to S606.

[0061] In S604, the CPU 201 flips the image data in the image file left to right. This is because when a mirror is used, the image is left to right reversed compared to the image that would be seen if the subject were actually observed. In S605, the CPU 201 associates left to right flip information indicating that the image has been flipped left to right with the image and records it. In this embodiment, this information is recorded as header information in the image file.

[0062] In S606, the CPU 201 determines whether the selected region information is the lower jaw. If it is the lower jaw, the process proceeds to S607; if it is not the lower jaw, the process proceeds to S609. As described above, when an image of the occlusal surface is captured using a mirror with a patient lying down, the dental arches of both the upper and lower jaws generally appear convex. Therefore, in the case of the lower jaw, in S607, the CPU 201 rotates the image data in the image file by 180 degrees so that the dental arch appears concave. Next, in S608, the CPU 201 associates 180-degree rotation information, which indicates that the image data has been rotated 180 degrees, with the image data and records it. In this embodiment, this information is recorded as header information in the image file.

[0063] In S609, the CPU 201 arranges and displays the five images on the display 204 in accordance with the selected region information. An example of the user interface at this time will be described with reference to Fig. 7. When the images have been displayed, the process ends.

[0064] [Image processing device user interface] 7 is a diagram showing an example of a user interface of the image processing device according to this embodiment. A method for arranging and displaying image data according to selected region information will be described with reference to FIG.

[0065] Reference numeral 700 denotes an end button, and when the user presses it, the CPU 201 determines that an end instruction has been issued by the user, and ends the series of processes. Reference numeral 701 denotes an upper jaw occlusal view display area. The CPU 201 displays image data in this area, the image file of which selected region information is the upper jaw. Reference numeral 702 denotes a mandibular occlusal view display area. The CPU 201 displays image data in this area, the image file of which selected region information is the mandible.

[0066] Reference numeral 703 denotes a front view display area. The CPU 201 displays image data in this area, the image file of which selected part information is the front view. Reference numeral 704 denotes a left side view display area. The CPU 201 displays image data in this area, the image file of which selected region information is the left side. Reference numeral 705 denotes a right side view display area. The CPU 201 displays image data in this area, the image file of which selected region information is the right side.

[0067] Selected region information display areas 706, 707, 708, 709, and 710 are displayed by the CPU 201. If the selected region information set by the imaging device 108 is incorrect, clicking on this area with an input device such as a mouse may display, for example, a pop-up menu, allowing the information to be changed.

[0068] Mirror use information display areas 711, 712, 713, 714, and 715. The CPU 201 displays mirror use information corresponding to each image. If the mirror use information set by the image capture device 108 is incorrect, clicking on this area with an input device such as a mouse may display, for example, a pop-up menu so that the information can be changed.

[0069] Horizontally flipped icons 716, 717, 718, and 719 are displayed by the CPU 201 only when horizontally flipped information corresponding to each image is available.

[0070] Reference numeral 720 denotes a 180-degree rotation icon. The CPU 201 displays the icon 720 only when there is 180-degree rotation information corresponding to each image.

[0071] 721 is a mark indicating the presence of residual teeth. If information on the state of the teeth is estimated by analyzing the image, it can be superimposed on the image to assist in diagnosis. When the image is flipped left and right or rotated 180 degrees, the coordinate position of the image will also be flipped left and / or rotated 180 degrees.

[0072] 722 is a patient information display area, which displays the patient's name, sex, and age at the time of examination. In this embodiment, images displayed in display areas 701 to 705 are displayed on the condition that they are from the same patient and have the same examination date, and if these conditions are not met, the images are not displayed on the same screen.

[0073] An examination date and time display area 723. In this embodiment, it is assumed that the correct date and time are set in the imaging device, and the imaging date and time match the examination date and time.

[0074] An update button 724 is provided. If the user performs a change operation in the selected body part information display area or the mirror use information display area, the CPU 201 assumes that the user has performed a display update operation in response to the press of the update button 724, and re-executes the series of processes in Fig. 6 in accordance with the updated information. Also, if the image before horizontal flipping or rotation was displayed using the original image display button 725, pressing the update button 724 will re-execute the series of processes in Fig. 6.

[0075] When the user presses the display original image button 725, the CPU 201 assumes that the user has performed an operation to display the original image, and displays the image immediately after shooting, before any left-right flipping or rotation, in each image area. Alternatively, when the display original image button 725 is pressed, the button may be switched to display an edited image display button. By switching to display the edited image display button, it becomes possible to easily revert from the image immediately after shooting to the image after editing.

[0076] In this embodiment, image data in the image file is updated in steps S604 and S607 in accordance with the horizontal flip and 180-degree rotation. However, the present invention is not limited to this. Instead, the image data itself in the image file may be stored in association with the horizontal flip flag and 180-degree rotation flag. In this case, the image data loaded into RAM 203 may be horizontally flipped or rotated in accordance with the flag when the user interface shown in FIG. 7 is displayed. This reduces the processing time required to write the horizontal flip and 180-degree rotation information to the image file. In this case, the horizontal flip icons 716, 717, 718, and 719 and the 180-degree rotation icon 720 display corresponding icons depending on whether the horizontal flip or rotation is enabled or disabled at the time of display. Furthermore, when the display original image button 725 is pressed, the horizontal flip or rotation processing according to the flag is not performed when the image data is displayed.

[0077] As described above, according to the first embodiment, image data is flipped left and right and rotated 180 degrees based on the selected region information and mirror use information selected during imaging. This makes it possible to display and record images in a layout that matches how the patient appears when actually observed, enabling medical personnel to make diagnoses using appropriate images.

[0078] Furthermore, by displaying a set of images associated with a patient and the same examination date, it is possible to prevent errors in diagnosis and recording due to mismatches between patients and images.

[0079] Furthermore, by viewing the intraoral photograph while checking the selected site information and mirror use information selected at the time of imaging, if an error is noticed, it can be corrected. Then, by updating the display based on the corrected information, the error at the time of imaging can be corrected and recorded.

[0080] Furthermore, by flipping or rotating the information recorded in association with the image coordinates along with the image, medical personnel can appropriately utilize the information to aid in diagnosis.

[0081] In addition, by recognizing whether the image is unedited or has been edited by inverting or rotating, if you need to check the image immediately after taking it, you can check the image data before inverting or rotating by pressing the original image display button.

[0082] Furthermore, by storing mirror use information when photographing each part, medical institutions that use the same procedure for intraoral photography do not need to set the mirror each time photography is performed.

[0083] <Second embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, when image data photographed for each patient and stored does not have associated body part information and mirror use information stored, the image processing device 101 infers the dental formula number, tooth type, tooth condition, and body part, and performs left-right flipping and up-down rotation. Note that the device configuration of the dental system in the second embodiment is the same as that described in the first embodiment with reference to Figures 1 to 3, and therefore description thereof will be omitted.

[0084] In this embodiment, machine learning is used to infer the dental number, tooth type, tooth condition, and tooth location. However, any image processing that can produce similar results may be used, such as pattern matching processing, edge detection, or discrimination processing using shading or color information. In this embodiment, the regions are roughly classified into front, occlusal, lateral, and others. In this embodiment, images stored for each patient in the HDD 205 of the image processing device 101 are processed. Images captured by the imaging device 108 and images transmitted from an external device are stored in the HDD 205.

[0085] [Image processing based on inference results] FIG. 8 is a flowchart showing the process of estimating a region of an image and the image processing in this embodiment.

[0086] First, in S800, the CPU 201 reads five image files, each containing five images taken using the five-image method and stored for each patient, from the HDD 205 to the RAM 203. Next, in S801, the CPU 201 infers whether each image in the five image files read in S800 is a frontal, occlusal, lateral, or other view.

[0087] In S802, the CPU 201 classifies the five images according to the inference results. If the images were properly captured using the five-image method, one image will be classified as a frontal image, two images will be classified as an occlusal image, and two images will be classified as a lateral image. If a result consistent with the five-image method is not obtained in S803, the CPU 201 determines that the input images were not appropriate, displays a warning in S804, and suspends subsequent processing. If a result consistent with the five-image method is obtained, the process proceeds to S805.

[0088] In S805, the CPU 201 records the "front view" in association with the image data as part information of the image classified as the front view, and also infers the dental formula number, tooth type, and condition.

[0089] In S806, the CPU 201 determines whether the top and bottom of the image are appropriate. Whether the top and bottom are appropriate is determined by whether the upper jaw teeth are at the top of the image and the lower jaw teeth are at the bottom of the image. If it is determined that the top and bottom are appropriate, the process proceeds to S808. If it is determined that the top and bottom are inappropriate, the process proceeds to S807. For example, when photographing a patient lying down, if the photograph is taken from directly facing the patient's head, an image will be recorded that is upside down.

[0090] In S807, the CPU 201 rotates the image data in the RAM 203, whose top and bottom were determined to be inappropriate in S806, by 180 degrees, and then stores a 180-degree rotation flag in association with the image data.

[0091] Next, in S808, the CPU 201 determines whether or not there is evidence of inversion for each of the two images classified as occlusal surfaces. The presence or absence of evidence of inversion is determined by whether or not the photographing date and time recorded in the header of the image file matches the file update date and time. If they do not match, it is determined that there is evidence of inversion. If they match, it is determined that there is no evidence of inversion. If there is evidence of inversion, the process proceeds to S809. If there is no evidence of inversion, the process proceeds to S810.

[0092] In S809, the CPU 201 left-right inverts the image data in the RAM 203 that corresponds to the image without any trace of inversion of the occlusal surface, out of the two images classified as the occlusal surface. Then, the CPU 201 associates a left-right inversion flag with the image data and stores it. This is because, as mentioned above, images of the occlusal surface are generally taken using a mirror.

[0093] In S810, the CPU 201 infers the dental formula number and tooth state for the two images classified as occlusal surfaces. In this embodiment, the dental formula number and tooth state are inferred using object detection in machine learning, and the inference results are obtained as rectangular information corresponding to coordinates on the image. As a result of the inference, the CPU 201 treats the image in which the maxillary teeth appear as an occlusal view image of the maxilla, and the image in which the mandibular teeth appear as an occlusal view image of the mandibular, and records the site information in association with the image. Furthermore, because a mirror is always used for occlusal view images, mirror use information is also associated and recorded.

[0094] In S811, the CPU 201 determines whether the dental arch is convex despite the fact that teeth of the lower jaw were detected in S810. Whether the dental arch is convex is determined based on the arrangement of rectangles detected as teeth on the image. If it is determined that the dental arch is convex despite being the lower jaw, the process proceeds to S812. If it is determined that the dental arch of the lower jaw is concave, the process proceeds to S813.

[0095] In S812, the CPU 201 rotates the image data in the RAM 203, in which the dental arch is convex despite being the lower jaw, by 180 degrees on the occlusal plane, and stores a 180-degree rotation flag in association with the image data.

[0096] Next, in S813, the CPU 201 infers the dental formula number and the state of the teeth for each of the two images classified as lateral views.

[0097] Then, in S814, the CPU 201 compares the inference results of the presence or absence of teeth, metal prostheses, etc. with the inference results of the occlusal surface, assuming that the two images classified as the lateral surface were not taken using a mirror. In this case, the part of the image where the molars are on the right side and the front teeth are on the left side is considered to be the left surface, and the part of the image where the molars are on the left side and the front teeth are on the right side is considered to be the right surface, and compared with the inference results of the occlusal surface. The comparison results for each tooth assuming that the images were not taken using a mirror are stored in RAM 203.

[0098] Next, in S815, the CPU 201 reverses the dental formula numbers to compare the inferred results of the tooth state when the two images classified as the lateral surface are assumed to have been taken using a mirror with the inferred results of the occlusal surface, and stores the comparison results for each tooth when the images are assumed to have been taken using a mirror in the RAM 203.

[0099] In S816, the CPU 201 determines whether or not each tooth has been photographed using a mirror, using the matching results for each tooth stored in S812 and S813. Details of the determination made here will be described later with reference to FIG.

[0100] In S817, the CPU 201 combines the determinations for each tooth to determine whether the image was taken using a mirror, and determines whether the image was taken using a mirror, without a mirror, or whether the result is inconsistent or undecided. If the result is inconsistent or undecided, the process proceeds to S820. If the result is not inconsistent or undecided, the process proceeds to S818.

[0101] In S818, the CPU 201 determines which of the side view images is the right side and which is the left side based on the result of the comprehensive determination. At this time, if an image assumed to have been captured using a mirror is selected in S815, the image data of the two side views in the RAM 203 are flipped left to right. Then, a left-right flip flag is associated with the image data and stored, and the body part information and mirror use information are recorded. If an image assumed to have been captured without using a mirror is selected, the image data of the two side views is not flipped left to right, and the body part information and mirror use information are recorded.

[0102] In S819, the CPU 201 performs the process of Fig. 6 as appropriate for the front view, maxillary occlusal view, mandibular occlusal view, left lateral view, and right lateral view classified in the above-mentioned process, and displays them using the user interface shown in Fig. 7. Then, the series of processes ends.

[0103] On the other hand, in S820, the CPU 201 determines whether or not there is a contradiction. If there is a contradiction, the process proceeds to S821. If there is no contradiction, the process proceeds to S822.

[0104] In S822, the CPU 201 cannot determine whether a mirror is being used or not, so it issues a notification such as "No correction will be made because no distinctive difference between the left and right sides can be detected." For the two lateral view images, an image in which the molars are positioned on the right and the front teeth are positioned on the left side is designated as a left lateral view, and an image in which the molars are positioned on the left and the front teeth are positioned on the right side is designated as a right lateral view. It is also assumed that a mirror is not being used. The part information and mirror information are then recorded in association with the image data. Then, the process proceeds to S819.

[0105] On the other hand, in S821, since there is no consistent combination, the CPU 201 issues a notice such as "Images of other patients may be mixed in" and ends the series of processes. [Method of inferring the body part shown in the image] Figure 9 is a conceptual diagram showing an example of the classification results of inferred images. Figure 9(a) shows an image inferred to be the upper jaw, and Figure 9(b) shows an image inferred to be the lower jaw.

[0106] Figures 9(c) and 9(d) show images inferred as lateral views when a mirror is not used, with Figure 9(c) showing an image inferred as the right view and Figure 9(d) showing an image inferred as the left view. When a mirror is not used, the molars appear on the left and the front teeth appear on the right, so the image shown in Figure 9(c) is assumed to be an image of the right view. When a mirror is not used, the front teeth appear on the left and the molars appear on the right, so the image shown in Figure 9(d) is assumed to be an image of the left view.

[0107] Figures 9(e) and 9(f) show images inferred to be side views when a mirror is used. Figure 9(e) is an image inferred to be the right view, which is the image of Figure 9(d) flipped left and right. When a mirror is used, the molars appear on the left and the front teeth appear on the right, so the image shown in Figure 9(e) is assumed to be an image of the right view. Figure 9(f) is an image inferred to be the left view, which is the image of Figure 9(c) flipped left and right. When a mirror is used, the front teeth appear on the left and the molars appear on the right, so the image shown in Figure 9(f) is assumed to be an image of the left view.

[0108] 10, 11, and 12 are tables showing tooth states by associating the inference results of the region with the dental formula number. Note that tooth states are associated when the coordinates on the image that are the inference results and the coordinates on the image that are the inference results of the tooth corresponding to the dental formula number overlap.

[0109] Figure 10(a) is a table showing an example of the inference results for the states of teeth 901 to 914 in the image of the upper jaw shown in Figure 9(a). Tooth 907 in the image of Figure 9(a) corresponds to dental almanac number uR1 in Figure 10(a), and tooth 901 corresponds to dental almanac number uR7. Tooth 908 corresponds to dental almanac number uL1, and tooth 914 corresponds to dental almanac number uL7. Reference numeral 915 denotes a characteristic part of the oral cavity known as the palatine fold, which is found only in the upper jaw.

[0110] Figure 10(b) is a table showing an example of the inference results for the states of teeth 916 to 927 in the image of the lower jaw shown in Figure 9(b). Tooth 922 in the image shown in Figure 9(b) corresponds to dental formula number bR1 in Figure 10(b), and tooth 916 corresponds to dental formula number bR7. Tooth 923 corresponds to dental formula number bL1, and tooth 927 corresponds to dental formula number bL5. 928 is the tongue. 929 is a characteristic part found only on the lower jaw called the lingual frenulum.

[0111] The dental almanac number and the inference result are stored as shown in Fig. 10. In this embodiment, if a tooth corresponding to the dental almanac number cannot be detected by inference from the image, it is treated as missing.

[0112] For example, in Figure 10(a), the state of tooth 907 with dental formula number uR1 is shown to be sound, and the state of tooth 906 with dental formula number uR2 is shown to be missing. Dental formula numbers uR3 to uR7 correspond to teeth 905 to 901, and dental formula numbers uL1 to uL7 correspond to teeth 908 to 914. The teeth corresponding to dental formula numbers uR8 and uL8 cannot be detected, so they are treated as missing. The same is true for Figure 10(b).

[0113] FIG. 11(a) is a table showing an example of the inference results for the states of teeth 930 to 944 in the image of the right side shown in FIG. 9(c) when a mirror is not used. Tooth 936 in the image of FIG. 9(c) corresponds to dental almanac number uR1 in FIG. 11(a), and tooth 930 corresponds to dental almanac number uR7. Tooth 942 corresponds to dental almanac number bR1, and tooth 938 corresponds to dental almanac number bR5. Because inference is performed assuming that FIG. 9(c) is an image of the right side, the resulting dental almanac numbers in FIG. 11(a) are all teeth that appear on the patient's right side. That is, tooth 937 corresponds to dental almanac number uL1, tooth 944 corresponds to dental almanac number bL2, and tooth 943 corresponds to the tooth on the left side of dental almanac number bL1, and therefore are not shown in the table of FIG. 11(a). Teeth 945 and 946 are teeth that appear outside the dentition and may occasionally be reflected in the subject image when a mirror is used.

[0114] 11(a), for example, the state of the tooth 936 with the dental formula number uR1 is shown to be healthy. Note that the teeth corresponding to the dental formula numbers uR8, bR6, bR7, and bR8 cannot be detected, and are therefore treated as missing.

[0115] FIG. 11(b) is a table showing an example of the inference results for the states of teeth 947 to 961 in the image of the left side shown in FIG. 9(d) when a mirror is not used. Tooth 947 in the image of FIG. 9(d) corresponds to dental almanac number uR1 in FIG. 11(b), tooth 948 corresponds to dental almanac number uL1, and tooth 953 corresponds to dental almanac number uL7. Tooth 954 corresponds to dental almanac number bR1, tooth 955 corresponds to dental almanac number bL1, and tooth 961 corresponds to dental almanac number bL7. Since inference is performed assuming that FIG. 9(d) is an image of the left side, the resulting dental almanac numbers in FIG. 11(b) are all only teeth that appear on the patient's left side. Note that teeth 962 and 963 are teeth that appear outside the dentition and may occasionally be reflected in the subject image when a mirror is used.

[0116] 11(b), for example, the state of the tooth 948 of the dental formula number uL1 is shown to be healthy. Note that the teeth corresponding to the dental formula numbers uL2, uL8, and bL8 cannot be detected, and are therefore treated as missing.

[0117] FIG. 12(a) is a table showing an example of the inference results for the state of teeth 964 to 978 in the image of the right side of FIG. 9(e) when a mirror is used, where L (left) in the dental formula numbers of the inference results shown in FIG. 11(b) is replaced with R (right). Tooth 970 in the image of FIG. 9(e) corresponds to dental formula number uL1 in FIG. 12(a), tooth 969 corresponds to dental formula number uR1, and tooth 964 corresponds to dental formula number uR7. Tooth 971 corresponds to dental formula number bL1, tooth 972 corresponds to dental formula number bR1, and tooth 978 corresponds to dental formula number bR7. Because inference is performed assuming that FIG. 9(e) is an image of the right side, the resulting dental formula numbers in FIG. 12(a) only include teeth that appear on the patient's right side. Teeth 979 and 980 are teeth that appear outside the dentition and may occasionally be reflected in the subject image when a mirror is used.

[0118] 12(a), for example, the state of tooth 969 of dental formula number uR1 is shown to be healthy. Note that teeth corresponding to dental formula numbers uR2, uR8, and bR8 cannot be detected and are therefore treated as missing.

[0119] FIG. 12(b) is a table showing an example of the inference results for the states of teeth 981 to 995 in the image of the left side shown in FIG. 9(f) when the mirror is used, where the R (right) of the dental formula numbers in the inference results shown in FIG. 11(a) is changed to L (left). Tooth 981 in the image of FIG. 9(f) corresponds to dental formula number uR1 in FIG. 12(b), tooth 982 corresponds to dental formula number uL1, and tooth 988 corresponds to dental formula number uL7. Tooth 995 corresponds to dental formula number bR2, tooth 994 corresponds to dental formula number bR1, tooth 993 corresponds to dental formula number bL1, and tooth 989 corresponds to dental formula number bL5. Because inference is performed assuming that FIG. 9(f) is an image of the left side, the resulting dental formula numbers in FIG. 12(b) only include teeth that appear on the patient's left side. Note that teeth 996 and 997 are teeth that appear outside the dental arch, and may occasionally be reflected in the subject image when a mirror is used.

[0120] In the example shown in Figure 12(b), for example, the state of the tooth 982 with the dental number uL1 is shown to be healthy. dental formula The teeth corresponding to numbers uL8, bL6, bL7, and bL8 cannot be detected and are therefore treated as missing.

[0121] [Explanation of overall evaluation] Next, a method for determining whether a mirror is used for the lateral view based on the above-mentioned inferences about the occlusal and lateral surfaces will be described.

[0122] Figure 13 is a table showing the matching state of the tooth condition on the occlusal surface and the side surface, and the pattern of the result estimated from the state. By referring to this table, it is judged whether each side view is the right image or the left image for each tooth with the same dental formula number shown in Figures 10 to 12. As shown in Figure 13, the combinations can be judged regardless of matching or mismatching (indicated by *), and the cases can be divided into a total of 11 patterns, and each teethThe matching state of the tooth condition of the formula number corresponds to one of 11 patterns. The determination method in S816 of Fig. 8 will be described below with reference to Fig. 13.

[0123] In FIG. 13, "Image 1" and "Image 2" indicate captured images of the side view. If Image 1 is the right side without flipping the image, then "Assumed right" in Image 1 corresponds to the image assumed when a mirror is not used (e.g., FIG. 9(c)). Also, "Assumed left" in Image 1 corresponds to the image assumed when a mirror is used (e.g., FIG. 9(f)). Similarly, if Image 2 is the left side without flipping the image, then "Assumed right" in Image 2 corresponds to the image assumed when a mirror is used (e.g., FIG. 9(e)), and "Assumed left" in Image 2 corresponds to the image assumed when a mirror is not used (e.g., FIG. 9(d)).

[0124] Then, the state of the teeth in the cases where images 1 and 2 are "assumed to be right" and "assumed to be left" are compared with the state of the teeth in the occlusal view with the same dental formula number, and it is determined for each tooth whether images 1 and 2 are assumed to be right images or left images.

[0125] ○ indicates that the state of the tooth with the same dental formula number in the occlusal view of Image 1 and Image 2 is consistent, × indicates that there is a misalignment. Also, * indicates that there is no relation to alignment or misalignment (either alignment or misalignment is acceptable).

[0126] For example, when image 1 is assumed to be the right (e.g., FIG. 9(c)), the tooth condition of dental formula number uR1 shown in FIG. 11(a), which shows the estimation result, is "sound," and the tooth condition of dental formula number uR1 shown in the corresponding occlusal view of FIG. 10(a), is also "sound," and they match (◯). When image 1 is assumed to be the left (e.g., FIG. 9(f)), the tooth condition of dental formula number uL1 shown in FIG. 12(b), which shows the estimation result, is "sound," and the tooth condition of dental formula number uL1 shown in the corresponding occlusal view of FIG. 10(a), is also "sound," and they match (◯). When image 2 is assumed to be the right (e.g., FIG. 9(e)), the tooth condition of dental formula number uR1 shown in FIG. 12(a), which shows the estimation result, is "sound," and the tooth condition of dental formula number uR1 shown in the corresponding occlusal view of FIG. 10(a), is also "sound," and they match (◯). Furthermore, if we assume that Image 2 is the left (for example, Figure 9(d)), the state of the tooth with dental formula number uL1 shown in Figure 11(b), which shows the estimation result, is "sound", and the state of the tooth with dental formula number uL1 shown in the corresponding occlusal view in Figure 10(a) is also "sound", so they match (◯). In this case, it corresponds to Pattern 1 in Figure 13, and the determination of whether Image 1 and Image 2 are right images or left images remains undecided.

[0127] Similarly, the state of the tooth with dental formula number uR7 shown in Figure 11(a), which shows the estimation result when image 1 is assumed to be the right, is "silver prosthesis", while the state of the tooth with dental formula number uR7 shown in Figure 10(a), which shows the corresponding occlusal view, is "silver full crown", which is inconsistent (x). On the other hand, Figure 11(a), which shows the estimation result when image 1 is assumed to be the left, is 12 The state of the tooth with dental formula number uL7 shown in (b) is "silver prosthesis", and the corresponding occlusal view of dental formula number uL 7 The state of the tooth in Figure 12(a), which shows the estimation results when Image 2 is assumed to be on the right, is also "silver prosthesis", which matches (◯). In addition, the state of the tooth with dental formula number uR7 shown in Figure 12(a), which shows the estimation results when Image 2 is assumed to be on the right, is "silver full crown", and the state of the tooth with dental formula number uR7 shown in Figure 10(a), which shows the corresponding occlusal view, is also "silver full crown", which matches (◯). On the other hand, Figure 12(a), which shows the estimation results when Image 2 is assumed to be on the left, is also "silver full crown", which matches (◯). 11 (b) Dental formula number uL 7 The condition of the teeth is "silver Full Crown" and the corresponding dental formula number uL shown in Figure 10(a) of the occlusal view. 7 Teeth condition teeth "Silver prosthesis" Inconsistency (×) In this case, the pattern in Figure 13 7 Image 1 is determined to be the right image and image 2 is determined to be the left image.

[0128] In this way, similar judgments can be made using the table in Figure 13. Do This is done for all teeth, and based on the tooth condition of each dental formula number in Image 1 and Image 2 and the corresponding tooth condition in the occlusal view, it is determined whether Image 1 and Image 2 correspond to "Undecided", "Right", "Left", or "Conflict" as shown in the judgment column in Figure 13. Then, the obtained judgment results are integrated to determine whether Image 1 and Image 2 correspond to "Right", "Left", or "Conflict". surface Specifically, the number of times image 1 is judged to be the right (or left) is compared with the number of times image 2 is judged to be the right (or left), and the image with the greater number of times judged to be the right (or left) is judged to be the right (or left) face, and the other image is judged to be the left (or right) face.

[0129] However, if there are many "undecided" or "conflicting" results, or if, for example, the number of times image 1 is determined to be on the right and the number of times it is determined to be on the left are the same or nearly the same, it is not possible to determine whether it is the right or left side (NO in S817). In such cases, in S820, it is determined whether the determination results are contradictory or whether a determination cannot be made because there are many undecided results, and if there is a contradiction, the process proceeds to S822, and if not, the process proceeds to S821.

[0130] In this embodiment, S80 7 , S80 9 , S81 2 In the above, the image data in the RAM 203 is inverted or rotated and stored in association with a flag, but the image data in the image file may be inverted and rotated and rewritten. The display process by updating the flag or the image file itself is the same as the method described in the first embodiment.

[0131] As described above, according to the second embodiment, even if the image is not stored in association with body part information and mirror use information, the body part indicated by the image can be accurately estimated and left-right flip processing and up-down rotation processing can be performed.

[0132] <Modification> In the second embodiment described above, the state of each tooth is estimated and the estimated result is analyzed to estimate the part shown in the image. In contrast to this, in this modified example, an example of a different estimation method will be described.

[0133] 14 is a conceptual diagram showing the detection results of characteristic tissues and instruments other than teeth. 1401 is the palatine fold, 1402 is the tongue, and 1403 and 1404 are the molars.

[0134] In the second embodiment, in S810, whether it is the upper or lower jaw is determined from the inference result of the teeth, but it may be determined that it is the upper jaw when a palatal fold 1401 is detected. Also, it may be determined that it is the lower jaw when a tongue 1402 or a lingual frenulum 929 shown in FIG. 9(b) is detected.

[0135] In addition, in S813, the state of the teeth in the lateral image is inferred, and in S816, a comprehensive judgment is made to determine whether the image is of the right or left side, but if the corner hooks 1403 and 1404 are present, it is possible to record information about mirror use as no mirror was used. If the image file's shooting date and time match the file's update date and time, it is possible to determine whether the image is of the right or left lateral view based on the positions of the front and back teeth without flipping the image left and right.

[0136] 9, if there are teeth 945, 946, 979, 980 that appear outside the dental arch, it may be determined that a mirror was used, and mirror use information may be recorded. If the image file's shooting date and time match the file's update date and time, the image may be flipped left and right, and whether it is a right lateral view or a left lateral view may be determined based on the relative positions of the front teeth and back teeth.

[0137] In the second embodiment, an example was described in which the dental formula number and the state of the teeth were precisely matched on the occlusal and lateral surfaces. However, if it is a reliable assumption that the images are a set of images taken inside the oral cavity of a single patient, matching may be determined based on the presence or absence of teeth or on characteristic teeth such as full crowns. Matching may also be determined based on tooth type, such as lateral incisors, canines, and molars, which is less precise than the dental formula number. Even if the accuracy of inference is not that high, the images can be flipped left and right or rotated 180 degrees to display them in a layout that matches how they appear when observing the patient.

[0138] As described above, according to this modified example, even if area information or mirror use information is not recorded at the time of imaging, the area indicated by the image can be estimated based on the type and condition of the teeth, the detection results of characteristic areas and instruments in the oral cavity, etc.

[0139] <Other embodiments> The present invention is based on a plurality of functions. Vessel? Even if applied to a system consisting of multiple devices, Place May be applied.

[0140] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0141] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0142] 101: Image processing device, 102: Network, 103: Dental electronic medical record display terminal, 104: Dental electronic medical record system, 105: Dental information DB, 106: In-hospital system, 107: Patient information DB, 108: Imaging device, 201: CPU, 202: ROM, 203: RAM, 204: Display, 206: Input device, 207: I / F

Claims

1. An acquisition means for acquiring an image file including first image data of an intraoral image, site information indicating a site in the oral cavity, and mirror use information indicating whether a mirror was used when photographing; a processing means for determining whether or not at least one of left-right flip processing and up-down rotation processing is necessary for the first image data of the acquired image file based on the part information and the mirror use information, and for performing the processing determined to be necessary.

1. An image processing device comprising:

2. The image processing device according to claim 1, wherein the site information added to the first image data of the intraoral image is one of the front, upper jaw, lower jaw, right side, and left side in the five-image method.

3. 3. The image processing apparatus according to claim 2, wherein the processing means performs the up-down rotation processing on the first image data when the part information is the lower jaw.

4. 4. The image processing device according to claim 1, wherein the processing means performs the left-right flip processing on the first image data when the mirror use information indicates that a mirror has been used.

5. 5. The image processing device according to claim 1, wherein when the first image data is processed by the processing means, second image data obtained by the processing is recorded in the image file.

6. 5. The image processing device according to claim 1, wherein when the first image data is processed by the processing means, a flag indicating the content of the processing is added to the first image data and recorded.

7. the image processing device further comprises a display means; An image processing device as described in any one of claims 1 to 6, characterized in that if the first image data of a corresponding part based on the part information or second image data obtained by processing the first image data by the processing means is present in a display area of ​​the display means predetermined according to the part, the second image data is displayed as third image data.

8. displaying, on the display means, at least one of the body part information and the mirror use information of the first image data corresponding to the third image data in association with the third image data; 8. The image processing apparatus according to claim 7, further comprising an input means for changing the body part information and the mirror use information displayed on the display means.

9. further comprising a selection means for selecting the first image data before being processed by the processing means; 9. The image processing apparatus according to claim 7, wherein when the first image data is selected by the selection means, the first image data is displayed on the display means.

10. 10. The image processing device according to claim 7, further comprising: displaying, on said display means, patient information and a photographing date associated with said first image data corresponding to said third image data.

11. An imaging system including an imaging device and an image processing device, The imaging device An imaging means for capturing an intraoral image; a setting means for setting site information indicating a site in the oral cavity and mirror use information indicating whether a mirror is used during imaging; a transmitting means for transmitting an image file in which the site information and mirror use information set at the time of photographing are added to the first image data of each intraoral image photographed by the imaging means to the image processing device, The image processing device receiving means for receiving the image file from the imaging device; a processing means for determining whether or not at least one of left-right flip processing and up-down rotation processing is necessary for the first image data of the received image file based on the body part information and the mirror use information, and for performing the processing determined to be necessary. An imaging system characterized by:

12. An acquisition means for acquiring image data of a plurality of intraoral images; A determination means for determining whether or not site information indicating a site in the oral cavity and mirror use information indicating whether a mirror was used or not when photographing are added to the image data of each intraoral image acquired by the acquisition means; When the site information and the mirror use information are not added, an estimation means for estimating the site and whether or not the mirror is used based on the features of each of the intraoral images; a storage means for adding site information and mirror use information to the image data of each of the intraoral images based on the site estimated by the estimation means and whether or not the mirror was used, and storing the result; 1. An image processing device comprising:

13. 13. The image processing device according to claim 12, further comprising a processing means for determining whether at least one of left-right flip processing and up-down rotation processing is necessary for the image data stored in the storage means based on the body part information and the mirror use information, and performing the processing determined to be necessary.

14. The estimation means a first estimation of a site indicated by each of the intraoral images among a plurality of predetermined intraoral sites based on a feature of each of the intraoral images; a second estimation of whether a mirror was used when capturing at least one of the intraoral images based on the state of each tooth in the plurality of intraoral images estimated based on the estimated location; and 14. The image processing device according to claim 12, wherein the image processing device performs the following.

15. The site is any one of the front, upper jaw, lower jaw, right side, and left side in the five-plate method, In the second estimation, the estimation means Estimating the state of each tooth from the intraoral images estimated to be the upper jaw and the lower jaw, estimating the state of each tooth from the intraoral images estimated to be the right surface and the left surface when it is assumed that a mirror was not used during photography, and estimating the state of each tooth from the intraoral images estimated to be the right surface and the left surface when it is assumed that a mirror was used during photography; 15. The image processing device of claim 14, wherein whether or not a mirror was used when the intraoral images estimated to be the right and left surfaces were taken is estimated based on the consistency between the state of each tooth of the upper and lower jaw and the state of each tooth on the right and left surfaces when a mirror was assumed not to have been used when the images were taken, and the consistency between the state of each tooth of the upper and lower jaw and the state of each tooth on the right and left surfaces when a mirror was assumed to have been used when the images were taken.

16. The image processing device according to claim 12 or 13, wherein the estimation means estimates the region indicated by each of the intraoral images from among a plurality of predetermined intraoral regions based on the features of each of the intraoral images.

17. The image processing device according to claim 12, 13, or 16, wherein the estimation unit estimates the location of each of the intraoral images based on a detection result of tissue other than teeth in each of the intraoral images.

18. The image processing device according to claim 12 , wherein the estimation unit estimates the location of each of the intraoral images based on a detection result of the type of tooth in each of the intraoral images.

19. The image processing device according to any one of claims 12, 13, and 16, characterized in that the estimation means estimates whether a mirror was used when each of the intraoral images was taken based on a detection result of a predetermined instrument in each of the intraoral images.

20. The image processing device according to any one of claims 12, 13, and 16, characterized in that the estimation means estimates whether a mirror was used when each of the intraoral images was taken based on a detection result of whether or not teeth outside the dentition are reflected in each of the intraoral images.

21. An acquisition step in which an acquisition means acquires an image file including image data of the intraoral image, site information indicating a site in the oral cavity, and mirror use information indicating whether a mirror was used at the time of imaging; a processing step in which a processing means determines whether or not at least one of left-right flip processing and up-down rotation processing is necessary for the image data of the image file based on the body part information and the mirror use information, and performs the processing determined to be necessary; An image processing method comprising:

22. In the imaging device, An imaging step of capturing an intraoral image; a setting step of setting site information indicating a site in the oral cavity and mirror use information indicating whether a mirror is used or not when photographing; a generating step of generating an image file in which the site information and mirror use information set at the time of photographing are added to the image data of each of the photographed intraoral images. In the image processing device, a processing step of determining whether or not at least one of left-right flip processing and up-down rotation processing is necessary for the image data of the image file generated in the generating step, based on the part information and the mirror use information, and performing the processing determined to be necessary. An image processing method comprising:

23. An acquisition step in which an acquisition means acquires image data of a plurality of intraoral images; A determination step in which the determination means determines whether or not site information indicating a site in the oral cavity and mirror use information indicating whether a mirror was used when photographing are added to the image data of each acquired intraoral image. An estimation step in which, when the site information and the mirror use information are not added, an estimation means estimates the site and whether the mirror is used or not based on features of each of the intraoral images; a storage step in which a storage means adds site information and mirror use information to the image data of each intraoral image based on the site estimated in the estimation step and whether or not the mirror was used, and stores the image data in a storage medium; An image processing method comprising:

24. A program for causing a computer to execute each step of the method according to any one of claims 21 to 23.

25. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 10 and 12 to 20.

26. A computer-readable storage medium storing the program according to claim 24 or 25.

Images