Image processing device, image processing method, and program

Abstract

To improve the accuracy of classifying the state of objects in an image. [Solution] The image processing device includes a first detection means for detecting regions of multiple elements that constitute an object from an object in an image, an acquisition means for determining feature quantities of the regions of the multiple elements, and a classification means for classifying the state of the object based on the feature quantities of the regions of the multiple elements.

Description

Technical Field

[0001] The present invention relates to a technique for classifying the state of an object in an image.

Background Art

[0002] As a use case for classifying the state of an object in an image, for example, the determination of the state of teeth in dentistry can be cited. The determination of the state of teeth in dentistry is performed by a dentist through visual confirmation, and the determination results of the state of each tooth are recorded by manual input or writing. In order to reduce the workload and improve the classification accuracy of such work, as in Patent Document 1, the state of teeth is determined from a patient's oral image by machine learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, it is not possible to appropriately classify dental diseases based on the oral examination information standard code specifications established by the Japanese Dental Association. In the oral examination information standard code specifications, the classification of dental diseases differs depending on the positional relationship of the caries area, restoration area, and crown area in the teeth. Therefore, even if the state of the teeth is determined by machine learning, it may not be possible to obtain desirable classification accuracy.

[0005] The present invention has been made in view of the above problems, and its object is to realize a technique for improving the classification accuracy of the state of an object in an image.

Means for Solving the Problems

[0006] To solve the above problems and achieve the objective, the image processing apparatus of the present invention comprises: a first detection means for detecting regions of multiple elements constituting an object from an object in an image; an acquisition means for determining feature quantities of the regions of the multiple elements; and a classification means for classifying the state of the object based on the feature quantities of the regions of the multiple elements. [Effects of the Invention]

[0007] According to the present invention, the accuracy of classifying the state of objects in an image can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram illustrating the functional configuration of the image processing apparatus according to this embodiment. [Figure 2] A diagram illustrating an input image according to this embodiment. [Figure 3] A diagram illustrating the hardware configuration of the image processing device according to this embodiment. [Figure 4] A flowchart illustrating the control process of the image processing apparatus according to this embodiment. [Figure 5] A diagram illustrating the output image according to this embodiment. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] First, with reference to Figure 1, the configuration and functions of the image processing apparatus according to this embodiment will be described.

[0011] <Functional Configuration> Figure 1 is a block diagram illustrating the functional configuration of the image processing apparatus according to this embodiment.

[0012] Each function of the image processing device 100 in this embodiment is realized by the hardware of the information processing device 300, which will be described later, and application software (hereinafter referred to as "application") that runs on the information processing device 300. The application is assumed to have software for utilizing the basic functions of the OS installed on the information processing device 300. The OS of the information processing device 300 may also have software for realizing the control processing in this embodiment.

[0013] The image processing device 100 of this embodiment receives an oral cavity image as input, determines the state of each tooth in the oral cavity image, and outputs a determination result image 107 in which the determination results are superimposed on the oral cavity image.

[0014] The image processing device 100 includes an image input unit 101, an image processing unit 102, an operation input unit 103, a display unit 104, and an output unit 105.

[0015] The image input unit 101 receives an input image 106 containing the object to be detected and outputs it to the image processing device 100. In this embodiment, the input image 106 is an image of the patient's oral cavity.

[0016] The image processing unit 102 receives an oral cavity image from the image input unit 101 and classifies the state of objects in the input image. In this embodiment, the object is a tooth.

[0017] The operation input unit 103 receives user input and outputs operation information to the image processing unit 102.

[0018] The display unit 104 displays the input image 106, the judgment result image 107, the application's GUI (Graphical User Interface), and the like.

[0019] The output unit 105 outputs and saves a judgment result image 107 based on the classification result of the image processing unit 102 of the state of each tooth.

[0020] Next, the functional configuration of the image processing unit 102 will be described.

[0021] The attention area detection unit 121 detects an object in the input image as an attention area. In the present embodiment, an object detection process is performed to detect a rectangular attention area including teeth or remnants. The object detection process only needs to be able to acquire the positions of each tooth, and may be an element area detection process.

[0022] The element area detection unit 122 detects areas of two or more types of elements constituting the detected object from the detected object. In the present embodiment, an area detection process is performed to detect an area for each dental element included in each detected tooth. The dental elements include two or more types of areas among the crown part, dental caries, and restorations.

[0023] The feature amount calculation unit 123 calculates a feature amount from the areas of the detected elements. In the present embodiment, a feature amount related to the state of the tooth determined by the state classification unit 124 is calculated from the areas of each dental element.

[0024] The state classification unit 124 determines and classifies the state of the object based on the feature amounts of the areas of the detected elements. In the present embodiment, the state of each tooth is determined and classified based on the feature amounts of the areas of each element.

[0025] The image synthesis unit 125 generates a determination result image in which the classification result of the state of each tooth is superimposed on the input image.

[0026] Next, referring to FIG. 2, the input image input to the image processing apparatus 100 according to the present embodiment will be described.

[0027] FIG. 2 illustrates images of teeth on the occlusal surface of the lower jaw in the states of dental caries, residual roots, partial restoration, total restoration, and secondary caries.

[0028] Image 210 is an example image of a carious area (CO-C3) 222 where the tooth is untreated and the crown region 221 is present. Caries may also be classified as a combined CO-C3 progression.

[0029] Image 211 is an example of a tooth in an untreated state with a root remnant. The root remnant is in a C4 stage of caries progression, with the crown region 221 having collapsed and disappeared. Region 224 is the gingival region.

[0030] The difference between caries (CO-C3) and root remnants is that caries (CO-C3) has a crown region, while root remnants do not. Because the classification of each tooth's condition changes depending on whether or not even a small portion of the crown region is present, it is difficult to determine such conditions using machine learning-based object detection or semantic segmentation.

[0031] In contrast, in this embodiment, two types of elements, the crown region 221 and the caries region 222, which are dental elements, are detected from the image 210, and the accuracy of the determination is improved by determining the state of the tooth in a rule-based manner based on the feature quantities calculated from the region of each element. Here, the caries region is defined as a part or the whole of the tooth where the tooth structure has been dissolved by acid produced by bacteria in the mouth.

[0032] Image 212 is an example image of a tooth that has been partially restored by treatment. Image 213 is an example image of a tooth that has been fully restored by treatment. The restoration area 223 is, for example, a metal restoration. However, it may also be a non-metallic restoration. The difference between partial and full restoration is that in partial restoration, the crown area 221 exists, but in full restoration, the crown area 221 does not exist. In this embodiment, the regions of two types of elements, the crown area 221 and the restoration area 223, which are dental elements, are detected from image 212, and the accuracy of the determination is improved by determining the state of the tooth in a rule-based manner based on the feature quantities calculated from the regions of each element.

[0033] Image 214 is an example image of a tooth in a secondary caries state. In this embodiment, to determine secondary caries, three types of regions are detected from Image 214: the crown region 221, the restoration region 223, and the caries region 222, which are dental elements. The accuracy of the determination is improved by determining the tooth condition using a rule-based method based on the feature quantities calculated from each region.

[0034] In this embodiment, the determination accuracy is improved by detecting the regions of two or more dental elements from the crown region, caries region, and restoration region from each tooth, and determining the state of each tooth based on the feature quantities calculated from the regions of each element.

[0035] <Hardware Configuration> Next, the hardware configuration of the image processing device 100 will be described with reference to Figure 3.

[0036] The information processing device 300 that realizes the functions of the image processing device 100 in this embodiment is a personal computer (desktop PC, notebook PC), a tablet PC, a smartphone, a cloud computer, etc.

[0037] The information processing device 300 includes a control device 301, an input device 302, an output device 303, and a storage device 304.

[0038] The control device 301 includes a processor (CPU) that performs calculation and control processing for the information processing device 300.

[0039] The storage device 304 includes a main memory 305 and an auxiliary storage device 306. The main memory 305 is a non-volatile memory (ROM) that stores programs executed by the processor, and a work memory (RAM) into which programs read from the ROM, constants and variables for executing the programs are loaded. The auxiliary storage device 306 is a large-capacity magnetic disk drive or SSD (Solid State Drive) that stores input images, output images, the OS (operating system), etc. The processor of the control device 301 controls each component of the information processing device 300 by loading programs stored in the ROM into the RAM and executing them, and functions as the image processing unit 102 of the image processing device 100. The control device 301 also controls the order in which each function of the image processing unit 102 is executed.

[0040] The input device 302 is an operating component (such as a keyboard or mouse) that accepts user input.

[0041] The output device 303 is a display device (such as a display) that shows images and information to the user.

[0042] The input device 302 corresponds to the operation input unit 103 in Figure 1. The output device 303 corresponds to the display unit 104 in Figure 1.

[0043] The information processing device 300 may have one control device 301 and one storage device 304. In other words, at least one processing unit (CPU) and at least one storage device are connected, and at least one processing unit executes a program stored in at least one storage device, thereby realizing the functions of the image processing unit 102 of the image processing device 100. Note that the control device and processing unit are not limited to CPUs, but may also be FPGAs, ASICs, etc.

[0044] <Image Processing> Next, the image processing according to this embodiment will be described with reference to Figure 4.

[0045] The process shown in Figure 4 is realized when the control device 301 of the information processing device 300 executes a program stored in the storage device 304 and functions as the image processing unit 102 of the image processing device 100.

[0046] In the image input processing of step S400, the control device 301 reads an oral cavity image from the storage device 304 as the image input unit 101.

[0047] In step S401, the control device 301, acting as a region of interest detection unit 121, performs object detection processing to detect teeth or remnants. The object detection processing acquires the position of each tooth, and since the crown of a tooth may be lost and only a root remains as caries progresses, the position of the root remains is also detected along with the tooth. Alternatively, the position of the teeth may be inferred by inputting image data into a learning model created by machine learning such as deep learning. Alternatively, the position of the teeth may be inferred using instance segmentation of machine learning. Instance segmentation can identify regions with shapes corresponding to objects. The position information of each tooth acquired by the object detection processing is used as a region of interest (ROI) when determining the state of each tooth.

[0048] Furthermore, inference processing can be performed by an image processing processor such as a GPU (Graphics Processing Unit). A GPU is a processor capable of performing a large number of multiply-accumulate operations and has the computational processing power to perform matrix operations of neural networks and other operations in a short time. In addition, the inference processing may be performed by the processor (CPU) of the control device 301 and the GPU working together, or by either the processor (CPU) of the control device 301 or the GPU.

[0049] In step S402, the control device 301, as an element region detection unit 122, performs region processing to detect regions for each dental element contained in each tooth detected in step S401. The region detection processing may be inferred using a semantic segmentation model created by machine learning. In this embodiment, the dental elements include the crown region 221, the caries region 222, and the restoration region 223 shown in Figure 2. The caries region 222 may be subdivided into C0-C3, etc., which are indicators representing the degree of caries progression. Furthermore, the dental elements to be determined may be elements based on the Oral Examination Information Standard Code established by the Japan Dental Association.

[0050] In step S403, the control device 301, acting as a feature calculation unit 123, calculates feature quantities related to the tooth state determined by the state classification unit 124 from the region of each dental element. Feature calculation is performed for each region of interest of each tooth calculated in step S401. In this embodiment, the crown boundary ratio q, restoration area ratio p, and secondary caries flag f are calculated as feature quantities.

[0051] The crown boundary ratio q is a feature that determines whether the state of each tooth is carious (C0-C3) or root remnant (C4) based on the detection results of dental elements. The crown boundary ratio q is defined as the ratio of the carious region to the crown region relative to the total length of the outer circumference of the carious region. For example, in image 210 of Figure 2, the entire outer circumference of the carious region 222 is in contact with the crown region 221, so the tooth in image 210 has q=1. Also, in image 211, the entire outer circumference of the carious region 222 is in contact with the gum region 224 and not with the crown region, so q=0. Note that the gum region 224 may also be detected.

[0052] Then, by setting the threshold qth for the crown boundary ratio q to qth = 0.1, if the crown boundary ratio q is less than or equal to the threshold qth, it can be determined that there is a root remnant, and if the crown boundary ratio q is greater than qth, it can be determined that there is caries (C0-C3).

[0053] The threshold value qth can be set arbitrarily. However, setting threshold qth=0 makes the determination more precise, but it may result in an error if the carious area is in contact with the crown area of ​​an adjacent tooth, so it is preferable that the threshold qth be greater than zero.

[0054] The restoration area ratio p is a feature that determines whether the condition of each tooth is a partial or complete restoration based on the detection results of dental elements. The restoration area ratio p is defined as the ratio of the area of ​​the restoration region to the total area of ​​the tooth of interest in the image.

[0055] Then, by setting the threshold pth for the area ratio p of the repaired material to 0.95, it can be determined that a full repair is performed if the area ratio p is greater than or equal to pth, and a partial repair is performed if the area ratio p is less than pth.

[0056] For example, in Figure 2, image 212 is determined to be a partial restoration because the proportion of the restored area 223 is less than the threshold pth of 0.95. In image 213, the proportion of the restored area 223 is greater than the threshold pth of 0.95, so it is determined to be a full restoration.

[0057] The threshold value pth can be set arbitrarily. However, setting threshold pth=1 will make the determination more precise, but there may be errors in determining the area of ​​interest, and an error may occur if the crown of an adjacent tooth is included. Therefore, it is desirable to set the threshold pth considering these factors.

[0058] The secondary caries flag f is a feature that determines whether or not the condition of each tooth is secondary caries based on the detection results of dental elements. The secondary caries flag f indicates whether or not there is contact between the carious area and the restoration area. It is determined to be True if the carious area and the restoration area are in contact, and False if they are not. For example, in image 214 of Figure 2, the restoration area 223 and the carious area 222 are in contact, so the secondary caries flag is True, and it is determined to be secondary caries. Secondary caries occurs when bacteria invade the gaps of restorations such as fillings and crowns used to treat caries, causing the tooth to become carious again, and this can be determined using the secondary caries flag.

[0059] Note that the features are not limited to the examples above; two or more types of features may be set and calculated depending on the state of the object being judged.

[0060] In the state classification process in step S404, the control device 301, acting as a state classification unit 124, determines the state of each tooth based on the feature quantities acquired in step S403.

[0061] In the image synthesis process of step S405, the control device 301, as the image synthesis unit 125, generates a judgment result image 107 by superimposing the classification results of the state of each tooth onto the input image 106. Alternatively, the judgment results may be generated as metadata instead of an image.

[0062] In the output processing of step S406, the control device 301 displays the judgment result image 107 generated in step S405 on the display unit 104 and outputs it from the output unit 105 to the storage device 304 for storage. Alternatively, the judgment result may be saved as metadata associated with the patient and transmitted to an external dental electronic medical record system, where the dental electronic medical record system interprets the metadata and displays the judgment result.

[0063] Next, with reference to Figure 5, the output image according to this embodiment will be described.

[0064] Figure 5 illustrates a determination result image 500 obtained by the image processing unit 102 according to this embodiment, which determines the condition of the teeth from images of the occlusal surface and the mandible.

[0065] The dashed area 501 indicates the area of ​​interest for each tooth. The state of each tooth is determined based on the features within the area of ​​interest.

[0066] The detection results of dental elements in the area of ​​interest are displayed superimposed on the input image as a crown detection area 521, a caries detection area 522, and a restoration detection area 523. The dental element detection areas may also be displayed using alpha blending.

[0067] The condition of each tooth is determined based on feature quantities calculated from the detection results of dental elements within the area of ​​interest.

[0068] The assessment results are displayed superimposed near each tooth, such as assessment label 530 "Caries (CO-C3)", assessment label 531 "Root Remnant", assessment label 532 "Partial Restoration", and assessment label 533 "Total Restoration". Note that the assessment label is omitted for healthy teeth.

[0069] As described above, this embodiment makes it possible to improve the accuracy of classifying the condition of teeth.

[0070] In this embodiment, an example in which oral images are used as the object of evaluation was described, but the invention is not limited to this example. For example, confectionery boxes may be packed by hand on a conveyor belt line, and this invention can be applied to inspecting the contents of such boxes.

[0071] The focus region detection unit 121 detects the confectionery as an object, and the element region detection unit 122 detects the type of confectionery as an element. By calculating a flag as a feature that determines whether the order of the confectionery is correct or not, it is possible to determine whether the confectionery is packed in the correct order based on the feature of each element.

[0072] Thus, this embodiment can be applied to use cases that match the process of calculating feature quantities from multiple elements detected from an object of interest and determining the state of the object based on the calculated feature quantities.

[0073] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0074] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0075] The disclosures herein include the following image processing apparatus, image processing methods, and programs. [Item 1] A first detection means for detecting the regions of multiple elements that constitute an object from an object in an image, An acquisition means for obtaining feature quantities of the regions of the aforementioned multiple elements, An image processing apparatus characterized by having a classification means for classifying the state of an object based on the feature quantities of the regions of the plurality of elements. [Item 2] Input means for inputting the aforementioned image and The image processing apparatus according to item 1, further comprising a second detection means for detecting an object in the aforementioned image. [Item 3] The image processing apparatus according to item 1 or 2, further comprising an image synthesis means for superimposing the classification results by the classification means onto the aforementioned image. [Item 4] The image processing apparatus according to item 2, characterized in that the second detection means detects a rectangular region containing the object by machine learning detection processing. [Item 5] The image processing apparatus according to item 2, characterized in that the second detection means detects a region with a shape corresponding to an object by machine learning instance segmentation. [Item 6] The aforementioned image is an oral cavity image, The aforementioned object is a tooth, The image processing apparatus according to any one of items 1 to 5, characterized in that the classification means classifies the condition of the teeth. [Item 7] The aforementioned element is a dental element, The image processing apparatus according to item 6, characterized in that the aforementioned dental elements include two or more types from the crown region, caries region, and restoration region. [Item 8] The image processing apparatus according to item 7, characterized in that the feature quantities are the crown boundary ratio, which is the ratio of the caries region to the crown region relative to the total length of the outer circumference of the caries region; the restoration area ratio, which is the ratio of the area of ​​the restoration region to the total area of ​​the tooth of interest; or a secondary caries flag, which indicates whether or not the caries region and the restoration region are in contact. [Item 9] The image processing apparatus according to item 7, characterized in that the state of the tooth includes caries in which the crown region exists or root remnants in which the crown region does not exist. [Item 10] The image processing apparatus according to item 7, characterized in that the condition of the tooth includes partial restoration or total restoration. [Item 11] The image processing apparatus according to item 7, characterized in that the condition of the tooth includes secondary caries. [Item 12] The detection means includes the step of detecting the regions of multiple elements that constitute an object from an object in an image, The acquisition means includes the step of determining the feature quantities of the regions of the plurality of elements, An image processing method characterized by comprising the step of classifying the state of an object based on the feature quantities of the regions of the plurality of elements. [Item 13] A program for causing a computer to function as one of the means of an image processing apparatus as described in items 1 through 11. [Explanation of Symbols]

[0076] 100...Image processing device, 101...Image input unit, 102...Image processing unit, 104...Display unit, 106...Input image, 107...Judgment result image, 121...Focus area detection unit, 122...Element area detection unit, 123...Feature quantity calculation unit, 124...State classification unit, 125...Image synthesis unit

Claims

1. A first detection means for detecting the regions of multiple elements that constitute an object from an object in an image, An acquisition means for obtaining feature quantities of the regions of the aforementioned multiple elements, An image processing apparatus characterized by having a classification means for classifying the state of an object based on the feature quantities of the regions of the plurality of elements.

2. Input means for inputting the aforementioned image and The image processing apparatus according to claim 1, further comprising a second detection means for detecting an object in the image.

3. The image processing apparatus according to claim 1, further comprising an image synthesis means for superimposing the classification results by the classification means onto the aforementioned image.

4. The image processing apparatus according to claim 2, characterized in that the second detection means detects a rectangular region including the object by machine learning detection processing.

5. The image processing apparatus according to claim 2, characterized in that the second detection means detects a region with a shape corresponding to an object by machine learning instance segmentation.

6. The aforementioned image is an oral cavity image, The aforementioned object is a tooth, The image processing apparatus according to claim 1, characterized in that the classification means classifies the condition of the teeth.

7. The aforementioned element is a dental element, The image processing apparatus according to claim 6, characterized in that the dental elements include two or more types from among the crown region, the caries region, and the restoration region.

8. The image processing apparatus according to claim 7, wherein the feature quantities are the crown boundary ratio, which is the ratio of the caries region to the crown region relative to the total length of the outer circumference of the caries region; the restoration area ratio, which is the ratio of the area of ​​the restoration region to the total area of ​​the tooth of interest; or a secondary caries flag, which indicates whether or not the caries region and the restoration region are in contact.

9. The image processing apparatus according to claim 7, characterized in that the state of the tooth includes caries in which the crown region exists or root remnants in which the crown region does not exist.

10. The image processing apparatus according to claim 7, characterized in that the condition of the tooth includes partial restoration or total restoration.

11. The image processing apparatus according to claim 7, characterized in that the tooth condition includes secondary caries.

12. The detection means includes the step of detecting the regions of multiple elements that constitute an object from an object in an image, The acquisition means includes the step of determining the feature quantities of the regions of the plurality of elements, An image processing method characterized by comprising the step of classifying the state of an object based on the feature quantities of the regions of the plurality of elements.

13. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 11.

Images